TheDcoder

The project looks interesting, I once thought of making a package manager myself However I have to say that I dislike npm, and the way it managed packages per folder... I like my packages to be in a central location, being shared with other scripts as needed. I guess everyone has their own preferences. What I do like is how a GitHub repository can be served as a package (if I am understanding it correctly), it will be interesting to see how this project grows. Best of luck @genius257

Yeah, that's the important bit, and as my tests have shown, when you remove the bottleneck, CRC is the clear winner which is more than twice as fast as MD5. The current differences are minuscule, and perhaps it is just plain bad luck due to several factors (including a potentially bad implementation of CRC in fsum.exe), so you cannot rule out that MD5 is magically better "in the real world" In the end, CRC is your fastest horse, but in a muddy road the speed doesn't matter, horses will slip and there will be luck involved. @RTFC Proudly married since the inception of the AutoIt Chatbox

Well bud, I believe my test is valuable because it shows which algorithm is faster, thus simpler. Note that the algorithm does not change how it works, it always functions the same, both in laboratory and in the real world. Thanks, I knew you would see the merit in my suggestion. This discussion was started because I was a bit "bamboozled" by your benchmark, however now I understand why. It is indeed outside the box, good one. But it may not be much of a help... and it might make things worse as well. Yeah, all thanks to the "warm-up" phenomenon, this happens in almost all benchmarks, the first test is usually performs worse compared to when it is done in the middle. Not so fast, here is a real test with my real Seagate 5400RPM SATA disk: TheDcoder@arch /m/d/D/W/Windows 10> stat -c %b Win10_1909_EnglishInternational_x64.iso 10594256 TheDcoder@arch /m/d/D/W/Windows 10> time rhash --crc32 Win10_1909_EnglishInternational_x64.iso ; Generated by RHash v1.4.0 on 2020-09-07 at 15:32.28 ; Written by Kravchenko Aleksey (Akademgorodok) - http://rhash.sf.net/ ; ; 5424252928 17:49.07 2020-01-13 Win10_1909_EnglishInternational_x64.iso Win10_1909_EnglishInternational_x64.iso 6B7B6CB5 ________________________________________________________ Executed in 42.95 secs fish external usr time 3.42 secs 267.00 micros 3.42 secs sys time 1.49 secs 108.00 micros 1.49 secs TheDcoder@arch /m/d/D/W/Windows 10> time rhash --md5 Win10_1909_EnglishInternational_x64.iso d1f08aea37586702f6fbe2fe3ea8c3fd Win10_1909_EnglishInternational_x64.iso ________________________________________________________ Executed in 44.20 secs fish external usr time 8.33 secs 195.00 micros 8.33 secs sys time 1.73 secs 77.00 micros 1.73 secs TheDcoder@arch /m/d/D/W/Windows 10> time rhash --sha1 Win10_1909_EnglishInternational_x64.iso 31549049b73cef72456af0bff71f494e76e1f506 Win10_1909_EnglishInternational_x64.iso ________________________________________________________ Executed in 46.09 secs fish external usr time 6.01 secs 377.00 micros 6.01 secs sys time 1.84 secs 152.00 micros 1.84 secs TheDcoder@arch /m/d/D/W/Windows 10> time rhash --sha256 Win10_1909_EnglishInternational_x64.iso b7a4a0786ea9ed51ac7874490885400f9cf3bbc71ab3dad468282da25147a29e Win10_1909_EnglishInternational_x64.iso ________________________________________________________ Executed in 45.17 secs fish external usr time 12.78 secs 0.00 micros 12.78 secs sys time 1.84 secs 464.00 micros 1.84 secs And it looks like my disk is being the bottleneck, even the SHA1 and SHA256 cryptographic algorithms take the same amount of time as CRC and MD5. Perhaps my processor is too fast In fact, if you observe closely, SHA1 was one second slower than SHA256... and SHA1 is not even secure anymore. So really, we can't find out the ideal hashing algorithm if we use real world tests like these, because there will be hidden bottlenecks. In the end, all I want to say is that CRC32 is best because it was designed for the purpose of checking differences in files, as opposed to the other cryptographic algorithms, which are designed to test the integrity of the file, thus they will be slower. I rest my case

I think you are going about this in a wrong way, especially how you think it is "unreal". It is as real as the bottleneck in your USB My intention was only to compare the speed of the algorithms, and only that. My test was just a proof that CRC32 is faster than MD5, obviously there are other factors which will limit the speed in a certain situation, and if the algorithm does not get the data as fast as it can process them, then it is of course going to be slower... which is what happened in your case, the speed was limited by the disk transfer rate. For a "real" test, you can try comparing MD5 and CRC32 for the same file on your internal hard disk, this should eliminate the bottleneck you had in your previous test.

I don't follow, why is it meaningless to compare the speed of the algorithm? In fact, it is the only meaningful way to do any objective benchmark, otherwise you'd be clumping in all sorts of "real-world" factors which are out of the control and skew the results, just like what happened in your case, where MD5 and CRC32 took the same amount of time due to disk bottleneck . Meaningless and subjective meat you mean Impossible to do a good objective test, too many factors involved. It would differ greatly even on the same computer, so it is not worth persuing this. The best way is to use a DLL always when possible... but in this case, it looks like WinAPI MD5 Algo is less optimized, so bad luck.

I was trying to compare the speed of the algorithms by isolating other outside influences as much as possible, thus the abnormal conditions. Not in this case, but I have used CRC32 when moving large files between disks previously and it was faster than any other hashing algo. I am no crypto expert, but I think the results would be fairly linear, i.e time will be proportional to the size. I know why, it is because the WinAPI crypt functions are already loaded by your script and they don't have the overhead of calling a 3rd-party executable, which a whole-another thing, thus increasing the overhead, this will significantly impact your performance for small files, but for large files the overhead wouldn't be noticeable. Ah, that makes sense, that was the bottleneck .

It should be a lot faster for larger files... so I am a bit surprised, maybe the CPU isn't the bottleneck here . Out of curiosity I did my own frugal test: TheDcoder@arch /tmp> time rhash --md5 bigfile cd573cfaace07e7949bc0c46028904ff bigfile ________________________________________________________ Executed in 1.74 secs fish external usr time 1679.79 millis 198.00 micros 1679.59 millis sys time 62.06 millis 78.00 micros 61.98 millis TheDcoder@arch /tmp> time rhash --crc32 bigfile ; Generated by RHash v1.4.0 on 2020-09-05 at 17:54.26 ; Written by Kravchenko Aleksey (Akademgorodok) - http://rhash.sf.net/ ; ; 1073741824 17:51.23 2020-09-05 bigfile bigfile 5B64C2B0 ________________________________________________________ Executed in 724.48 millis fish external usr time 666.49 millis 271.00 micros 666.22 millis sys time 58.13 millis 107.00 micros 58.02 millis TheDcoder@arch /tmp> stat bigfile File: bigfile Size: 1073741824 Blocks: 0 IO Block: 4096 regular file Device: 30h/48d Inode: 42640 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/TheDcoder) Gid: ( 1000/TheDcoder) Access: 2020-09-05 17:54:27.682908480 +0530 Modify: 2020-09-05 17:51:23.904011423 +0530 Change: 2020-09-05 17:51:23.904011423 +0530 Birth: - CRC32 was ~2.4x faster than MD5 when tested with a 1 GB file, the file was read from RAM so the disk overhead should not exist.

Looks good! A potential improvement is to add support for CRC32 (same algorithm used by TeraCopy), it is much faster than MD5

I have posted a fixed version in that thread, I am going to try it with a new project, wish me luck guys!

@dany Thanks for the UDF, it looks very promising, sadly it is no longer working with the latest version of AutoIt. I am not going to necro-post this thread, but I am indeed here to resurrect it! I fixed up the UDF to run with the latest version, as well as converted the encoding of the script files to UTF-8 Here is the file: AU3Text (Fixed - UTF16 NoBOM - AutoIt v3.3.14.5).zip Edit: Fixed some more stuff. Edit 2: Fixed yet more stuff, reverted some of the code to use UTF16 w/o BOM because Ini functions require that encoding...

Thank you all for the input (sic), I always look the obvious functions like SetLimitText . I did a test and it looks promising: #include <EditConstants.au3> #include <GUIConstantsEx.au3> #include <GuiEdit.au3> #include <MsgBoxConstants.au3> #include <WindowsConstants.au3> Example() Func Example() GUICreate("My GUI edit", 500, 500) Local $idMyedit = GUICtrlCreateEdit("", 0, 0, 500, 500, $ES_AUTOVSCROLL + $WS_VSCROLL) GUISetState(@SW_SHOW) Local $sDumpString = "0123456789" Local $sPrevString While True While True GUICtrlSetData($idMyedit, $sDumpString, 1) Local $sCurrString = GUICtrlRead($idMyedit) If $sPrevString = $sCurrString Then ExitLoop $sPrevString = $sCurrString WEnd Local $iLimit = StringLen($sPrevString) ConsoleWrite("Edit buffer limit reached at " & $iLimit & " characters." & @CRLF) If MsgBox($MB_YESNO, "Buffer Limit", "Edit buffer limit reached at " & $iLimit & " characters. Do you want to continue?") = $IDNO Then ExitLoop $sDumpString = GUICtrlRead($idMyedit) _GUICtrlEdit_SetLimitText($idMyedit, $iLimit * 2) WEnd ; Loop until the user exits. While 1 Switch GUIGetMsg() Case $GUI_EVENT_CLOSE ExitLoop EndSwitch WEnd GUIDelete() EndFunc I was able to increase the limit up-to 960K characters, after that the GUI stopped responding when the script was attempting to insert another 960K characters. So hopefully we can keep track of the count and increase the limit as we go dynamically. Will implement this in my real script and report back in a few weeks

Hello everyone, Not too long ago I stumbled across the buffer limit in an edit control, at first I thought it must be a bug in my code, but after some investigation I quickly found out that the edit control by default has a limit of characters it can display... for me it can only show 30K characters, here is a reproducer script I wrote (by forking the example code): #include <EditConstants.au3> #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> Example() Func Example() GUICreate("My GUI edit", 500, 500) Local $idMyedit = GUICtrlCreateEdit("", 0, 0, 500, 500, $ES_AUTOVSCROLL + $WS_VSCROLL) GUISetState(@SW_SHOW) Local $sPrevString While True GUICtrlSetData($idMyedit, "0123456789", 1) Local $sCurrString = GUICtrlRead($idMyedit) If $sPrevString = $sCurrString Then ExitLoop $sPrevString = $sCurrString WEnd ConsoleWrite("Edit buffer limit reached at " & StringLen($sPrevString) & " characters." & @CRLF) ; Loop until the user exits. While 1 Switch GUIGetMsg() Case $GUI_EVENT_CLOSE ExitLoop EndSwitch WEnd GUIDelete() EndFunc Is there a way to increase or entirely remove this limit? How can I find out the buffer size of a certain edit control? Failing that, is there an efficient way to check if I have hit the buffer limit? I am sure many of you have come across this issue, and I am hoping for a good work around Thanks for the help in advance!

@rootx Glad to be of assistance

@rootx You have to go through each byte and compare the values with known values. Remember that a byte is just nothing but an 8 digit binary number My apologies for the lack of an example, I am a bit busy at the moment. Hopefully someone will take out the time to show you

Some clarification: I think @error is only reset after calling another function, not after every command (statement). So the first code example should also work in theory, however I do agree that it is bad style to use @error like that.

@monoceres Thank you for sharing! I am sure the project has immense value, I will make sure to study the code sometime to get ideas and inspirations

Like I expected, I overlooked something simple, I had the date in the wrong order in my example, I was using DMY instead of YMD. #include <DateTimeConstants.au3> #include <GUIConstantsEx.au3> #include <GuiDateTimePicker.au3> #include <MsgBoxConstants.au3> Example() Func Example() GUICreate("My GUI get date", 200, 200, 800, 200) Local $idDate = GUICtrlCreateDate("", 10, 10, 185, 20, $DTS_TIMEFORMAT) Local $hDate = GUICtrlGetHandle($idDate) ; to select a specific default format Local $sStyle = "HH:mm:ss" GUICtrlSendMsg($idDate, $DTM_SETFORMATW, 0, $sStyle) ; Set time Local $aDate[7] = [False, 2020, 1, 1, 13, 33, 37] Local $bRet = _GUICtrlDTP_SetSystemTime($hDate, $aDate) ConsoleWrite('_GUICtrlDTP_SetSystemTime returned ' & ($bRet ? 'success' : 'failure') & @CRLF) GUISetState(@SW_SHOW) ; Loop until the user exits. While GUIGetMsg() <> $GUI_EVENT_CLOSE WEnd MsgBox($MB_SYSTEMMODAL, "Time", GUICtrlRead($idDate)) EndFunc ;==>Example This works perfectly and it doesn't even depend on strings, let alone have regional issues! I am happy Edit: Just noticed another helpful tidbit, remember to set the first element of the time/date array to False, otherwise it won't work! At-least it did not for me.

@Danp2 You are right about having to include date along with the time, but like TheSaint has said, the format of date + time depends on the locale of the machine and I need my script to work in various locales, not only English-like ones. At this point I am pretty sure GUICtrlSetData only takes the region-specific string as valid input, but what stumps me is why _GUICtrlDTP_SetSystemTime also doesn't seem to work? @TheSaint Thanks for clarifying on my behalf. Depends on how you see it, if I compromise with not properly supporting all locales, I can definitely just hardcode the English string an use a fake date to set the time. But I'd rather not do that.

It is as simple as converting, but conversion is the hard part, and there is no standard reference that can be used, nor is there any Windows API that I can use to convert time and date between different locales... so practically impossible unless I limit myself to a specific version of Windows in specific regional settings. -- I did find a promising alternative function (_GUICtrlDTP_SetSystemTime) that I could use to set the time, but that too didn't work: #include <DateTimeConstants.au3> #include <GUIConstantsEx.au3> #include <GuiDateTimePicker.au3> #include <MsgBoxConstants.au3> Example() Func Example() GUICreate("My GUI get date", 200, 200, 800, 200) Local $idDate = GUICtrlCreateDate("1953/04/25", 10, 10, 185, 20, $DTS_TIMEFORMAT) ; to select a specific default format Local $sStyle = "HH:mm:ss" GUICtrlSendMsg($idDate, $DTM_SETFORMATW, 0, $sStyle) ; Set time Local $aDate[7] = [True, 0, 0, 0, 13, 33, 37] Local $hDate = GUICtrlGetHandle($idDate) Local $bRet = _GUICtrlDTP_SetSystemTime($hDate, $aDate) ConsoleWrite('_GUICtrlDTP_SetSystemTime returned ' & ($bRet ? 'success' : 'failure') & @CRLF) GUISetState(@SW_SHOW) ; Loop until the user exits. While GUIGetMsg() <> $GUI_EVENT_CLOSE WEnd MsgBox($MB_SYSTEMMODAL, "Time", GUICtrlRead($idDate)) EndFunc ;==>Example

@TheSaint Thanks, that clears about what $now is, but unfortunately the string returned by _Now is locale-specific, meaning it changes across different regions But I want to set the time in my format, not the PC's format...

@TheSaint Can you give more info on the $now parameter in your function? It looks like an array which contains the time that you set: $time = $now[2] ; ... GUICtrlSetData($Date_totime, $time)

@TheSaint Thanks for posting, but I am having issues with GUICtrlSetData where I attempt to set the time in the control, do you have code which sets data?

Hi everyone, I am bit stumped as to why I am not able to set the time in the Date control #include <DateTimeConstants.au3> #include <GUIConstantsEx.au3> #include <MsgBoxConstants.au3> Example() Func Example() GUICreate("My GUI get date", 200, 200, 800, 200) Local $idDate = GUICtrlCreateDate("1953/04/25", 10, 10, 185, 20, $DTS_TIMEFORMAT) ; to select a specific default format Local $sStyle = "HH:mm:ss" GUICtrlSendMsg($idDate, $DTM_SETFORMATW, 0, $sStyle) ; Set time Local $iRet = GUICtrlSetData($idDate, '13:33:37') ConsoleWrite('GUICtrlSetData returned ' & ($iRet = 1 ? 'success' : 'failure') & @CRLF) GUISetState(@SW_SHOW) ; Loop until the user exits. While GUIGetMsg() <> $GUI_EVENT_CLOSE WEnd MsgBox($MB_SYSTEMMODAL, "Time", GUICtrlRead($idDate)) EndFunc ;==>Example The documentation for GUICtrlSetData clearly mentions that it uses the same format as GUICtrlRead: But I get failure What gives? Thanks for the help in advance!

@Nine I see, a fully-fledged DB seems to be an over-kill, but it is an option nonetheless @MrCreatoR Nice, I came across it once but I wasn't able to find it after that.

@Nine Sounds interesting, by "database" I assume you mean an SQLite database?