Thread: Faster file checking c#

Results 1 to 5 of 5
  1. #1 Faster file checking c# 
    The internet's reject
    Andy's Avatar
    Join Date
    Jun 2007
    Age
    33
    Posts
    4,371
    Thanks given
    148
    Thanks received
    1,522
    Rep Power
    3133
    The naming in this is weird becasue i get moderated every time i mention the T word, replace the b in borrents with a t

    Last night I was very intoxicated and rewriting parts of my scraper for GetStrike as i noticed it was bottle necking.

    So GetStrike has millions of borrents indexed to disk for users to download; so this means while scraping sometimes I need to check if a file exist or not. Each folder contains about 2.8 million files on average, I should segment them more later.

    The structure of the folders is pretty straight forward.




    So we parse each folder to check if the file exist, doing anything like *.* would be slow to begin with,

    Here was the original causing the bottle neck

    Code:
     for (int i= 0; i < groupLength; ++i) {
               if (!File.Exist(dest + i)) {
                   Console.WriteLine("borrent didn't exist");
                   Downloadborrent(borrent_link, hash);
                   break;
               } else {
                   break;
               }
           }

    Too slow in in our case, average response was about 450ms

    Using Exists() to check for file or directory name in use is subject to race conditions. Not a lot of people realize this. After the scraper Exists() check has finished, another instance of the scraper could have created the borrent with that hash before the original instance code reaches the point where you download a borrent, for example. I found it faster to simply to open the borrent, specifying the FileShare parameter.

    Code:
    using (var stream = File.Open(borrentPath, FileMode.CreateNew, FileAccess.Write, FileShare.None))
            {
                // Write borrent
            }

    So handling the IOException on failure may result in code that is less prone to race conditions,:

    • If another instance has already created the borrent,
      Code:
       FileMode.CreateNew
      will cause an IOException to be thrown
    • If the original instance open and create succeeds, because of
      Code:
      FileShare.None
      , no other instance can access the file until it closes it.



    It seems it is not possible to check whether a file is currently in use, and not throw an exception, without PInvoke:



    Code:
    class Win32
        {
            const uint FILE_READ_DATA = 0x0001;
            const uint FILE_SHARE_NONE = 0x00000000;
            const uint FILE_ATTRIBUTE_NORMAL = 0x00000080;
            const uint OPEN_EXISTING = 3;
            const int INVALID_HANDLE_VALUE = -1;
    
            [DllImport("kernel32.dll", SetLastError=true)]
            internal static extern IntPtr CreateFile(string lpFileName,
                                                   uint dwDesiredAccess,
                                                   uint dwShareMode,
                                                   IntPtr lpSecurityAttributes,
                                                   uint dwCreationDisposition,
                                                   uint dwFlagsAndAttributes,
                                                   IntPtr hTemplateFile);
    
            [DllImport("kernel32.dll")]
            internal static extern bool CloseHandle(IntPtr hObject);
        }
    Then actually checking if its in use

    Code:
     bool IsFileInUse(string fileName)
        {
                IntPtr hFile = Win32.CreateFile(fileName, Win32.FILE_READ_DATA, 0, IntPtr.Zero, Win32.OPEN_EXISTING, Win32.FILE_ATTRIBUTE_NORMAL, IntPtr.Zero);
                if (hFile.ToInt32() == Win32.INVALID_HANDLE_VALUE)
                    return true;
    
                Win32.CloseHandle(hFile);
                return false;
        }
    This faster method is also prone to race conditions, unless you return the file handle from it, and pass that to the relevant
    Code:
    FileStream
    constructor.

    Total check time for 17 million files was 32ms and some change.

    Funny enough, if I don’t check anything at all and just download, the duplicate borrents and overwrite, its faster than checking if it exist. But I nonetheless the FileShare method works best for my needs.

    tl;dr Exist(); sucks
    What am I watching now?

    Reply With Quote  
     

  2. #2  
    Registered Member
    Optimum's Avatar
    Join Date
    Apr 2012
    Posts
    3,570
    Thanks given
    871
    Thanks received
    1,745
    Rep Power
    5000
    I'm not good with C#

    But I used async to open files

    Quote Originally Posted by DownGrade View Post
    Don't let these no life creeps get to you, its always the same on here. They'd rather spend hours upon hours in the rune-server spam section then getting laid! ha ha!Its honestly pathetic i haven't seen so many lowlifes in my life its actually insane i wish that this section would just vanish its probably the only way to get these people out of the community...
    PLEASE BE AWARE OF IMPOSTERS MY DISCORD ID: 362240000760348683
    Reply With Quote  
     

  3. #3  
    Registered Member
    Join Date
    Apr 2014
    Posts
    15
    Thanks given
    2
    Thanks received
    0
    Rep Power
    11
    #change torrent to borrent!


    string torrent = "torrent";

    if(torrent != "borrent")
    {
    torrent = "borrent";
    }
    [IMG][/IMG]
    Reply With Quote  
     

  4. #4  
    Registered Member
    Join Date
    Jul 2008
    Posts
    76
    Thanks given
    3
    Thanks received
    5
    Rep Power
    3
    Haha nice. might use it in VB.net some small changes sinse C# and VB are sisters
    Reply With Quote  
     

  5. #5  
    Deathlypvpz Founder

    Join Date
    Aug 2012
    Posts
    486
    Thanks given
    6
    Thanks received
    116
    Rep Power
    43
    noiceeeeee
    Reply With Quote  
     


Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. Exisintg file check?
    By _Patrick_ in forum Application Development
    Replies: 10
    Last Post: 09-03-2014, 05:09 PM
  2. File checking Loop, Rep + Thanks
    By Paradox_ in forum Requests
    Replies: 5
    Last Post: 04-05-2012, 07:13 AM
  3. Replies: 1
    Last Post: 08-01-2010, 09:47 AM
  4. [RS2HD][508] A faster Coordinate check
    By Itz Matthew ♥ in forum Snippets
    Replies: 4
    Last Post: 07-17-2009, 11:25 PM
  5. Replies: 7
    Last Post: 04-15-2009, 06:21 PM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •