I'm using berkeley db to store 100 milion keys and theirs values. There is one instance of berkeley db on the server. One writer adds and removes records and five readers (five independent applications) read proper values.
Sometimes one of my readers stucks and I have to kill it and run it again. I attached to it using gdb:
[Switching to LWP 100674 of process 54232]
0x0000000000886771 in __lock_ohash ()
(gdb) where
#0 0x0000000000886771 in __lock_ohash ()
#1 0x000000000087a1c4 in __lock_get_internal ()
#2 0x00000000008793d4 in __lock_vec ()
#3 0x000000000078e094 in __db_lget ()
#4 0x00000000007ff26d in __bam_search ()
#5 0x00000000007f07f6 in __bamc_search ()
#6 0x00000000007edb13 in __bamc_get ()
#7 0x0000000000783091 in __dbc_iget ()
#8 0x000000000078b793 in __dbc_get_pp ()
#9 0x0000000000762a5a in Dbc::get(Dbt*, Dbt*, unsigned int) ()
...
Why does it stuck in __lock_ohash() function? When it stucks the whole application freezes until I kill it. Is it a deadlock? Why database does not manage and solve it?
Version of berkeley db: 18.1.40
Flags used by writer and readers:
uint32_t flags = DB_INIT_LOCK | DB_INIT_MPOOL | DB_SYSTEM_MEM | DB_INIT_TXN |
DB_CREATE | DB_REGISTER;
OS: FreeBSD 13.0-RELEASE-p6
Shared memory info:
[adg#ovhs8 ~]$ ipcs -M
shminfo:
shmmax: 4294967296 (max shared memory segment size)
shmmin: 1 (min shared memory segment size)
shmmni: 192 (max number of shared memory identifiers)
shmseg: 128 (max shared memory segments per process)
shmall: 1048576 (max amount of shared memory in pages)
Database size - 4,6 GB:
[adg#ovhs8 ~/adg_db/Database/database]$ ls -ltrh
total 3349522
-rw-r----- 1 adg adg 10M Mar 13 23:11 log.0000000001
-rw-r----- 1 adg adg 4.6G Mar 14 09:26 adgdatabase.db
-rw-r----- 1 adg adg 24B Mar 14 09:29 __db.001
-rw-r----- 1 adg adg 200B Mar 14 09:30 __db.register
Related
I have a ciphered .docx document I would like to recover and I don't remember the password. I'm trying brute-forcing it but it's taking way too long, so it's not going to be option. However, I know the exact content of part of it (296 characters). Any help?
Unfortunately, part of the document wouldn't help.
To get to the cleartext, any cracker would still need to go through trying to crack the password hash that is exported from the document, and with your logic try to decrypt the file and interpret it's content, compare it to the known cleartext. There is no such funcitonality, especially for specialized document formats.
Here is an example how to approach it:
Document: encrypted_doc.docx
Password: 123horse123
You will have to use office2john to export the hash to be cracked from the document.
wget https://raw.githubusercontent.com/magnumripper/JohnTheRipper/bleeding-jumbo/run/office2john.py
python office2john.py encrypted_doc.docx > doc_pass_hash.txt
cat doc_pass_hash.txt
encrypted_doc.docx:$**office$*2013***100000*256*16*e77e386a8e68462d2a0a703718febbc9*08ee275ccf4946ae0e5922e9ff3114b7*0ab5fc00964f7ed4be9e45c77a33b441b2c4874d28e4bc30f38e99bfb169fcf4
Remembering some information about the password(complexity, some chosen words if any, character set etc.) mask attack could help you run a more effective way to uncover the document.
Run hashcat --help to see which document file are you dealing with:
9700 | MS Office <= 2003 $0/$1, MD5 + RC4 | Documents
9710 | MS Office <= 2003 $0/$1, MD5 + RC4, collider #1 | Documents
9720 | MS Office <= 2003 $0/$1, MD5 + RC4, collider #2 | Documents
9800 | MS Office <= 2003 $3/$4, SHA1 + RC4 | Documents
9810 | MS Office <= 2003 $3, SHA1 + RC4, collider #1 | Documents
9820 | MS Office <= 2003 $3, SHA1 + RC4, collider #2 | Documents
9400 | MS Office 2007 | Documents
9500 | MS Office 2010 | Documents
9600 | MS Office 2013 | Documents
Based on what you can recall from the password, you can choose from the following:
- [ Attack Modes ] -
# | Mode
===+======
0 | Straight
1 | Combination
3 | Brute-force
6 | Hybrid Wordlist + Mask
7 | Hybrid Mask + Wordlist
Here are the options for hashcat to specify the password:
?l = abcdefghijklmnopqrstuvwxyz
?u = ABCDEFGHIJKLMNOPQRSTUVWXYZ
?d = 0123456789
?h = 0123456789abcdef
?H = 0123456789ABCDEF
?s = «space»!"#$%&'()*+,-./:;<=>?#[\]^_`{|}~
?a = ?l?u?d?s
?b = 0x00 - 0xff
You can also create your own dictionary, which then will be used when generating the passwords, if you remember at least part of the password. This can be the most efficient help.
So in my example, let's run a brute force attack with mask(3 digits, 5 alphabetical characters, and another 3 digits):
hashcat -m 9600 -a 3 doc_pass_hash.txt --username -o cracked_pass.txt ?d?d?d?l?l?l?l?l?d?d?d --force
You can hit [s] for status:
[s]tatus [p]ause [b]ypass [c]heckpoint [q]uit => s
Session..........: hashcat
Status...........: Running
Hash.Type........: MS Office 2013
Hash.Target......: $office$*2013*100000*256*16*e77e386a8e68462d2a0a703...69fcf4
Time.Started.....: Sat May 30 16:59:30 2020 (3 mins, 41 secs)
Time.Estimated...: Next Big Bang (17614 years, 157 days)
Guess.Mask.......: ?d?d?d?l?l?l?l?l?d?d?d [11]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........: 21 H/s (7.50ms) # Accel:128 Loops:32 Thr:1 Vec:8
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 4608/11881376000000 (0.00%)
Rejected.........: 0/4608 (0.00%)
Restore.Point....: 0/1188137600000 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:9-10 Iteration:24672-24704
Candidates.#1....: 623anane123 -> 612kerin123
As you see, this one doesn't seem to be very effective (Time.Estimated...: Next Big Bang (17614 years, 157 days)), however, adding a wordlist is a good idea:
cat wordlist.txt
dog
horse
cat
hashcat -m 9600 -a 6 doc_pass_hash.txt wordlist.dict ?d?d?d?l?l?l?l?l?d?d?d --username -o cracked_pass.txt --forces
Session..........: hashcat
Status...........: Running
Hash.Type........: MS Office 2013
Hash.Target......: $office$*2013*100000*256*16*e77e386a8e68462d2a0a703...69fcf4
Time.Started.....: Sat May 30 17:15:34 2020 (1 min, 25 secs)
Time.Estimated...: Next Big Bang (734631 years, 226 days)
Guess.Base.......: File (wordlist.dict), Left Side
Guess.Mod........: Mask (?d?d?d?l?l?l?l?l?d?d?d) [11], Right Side
Guess.Queue.Base.: 1/1 (100.00%)
Guess.Queue.Mod..: 1/1 (100.00%)
Speed.#1.........: 2 H/s (0.47ms) # Accel:128 Loops:32 Thr:1 Vec:8
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 129/35644128000000 (0.00%)
Rejected.........: 0/129 (0.00%)
Restore.Point....: 0/3 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:43-44 Iteration:32000-32032
Candidates.#1....: dog360verin123 -> cat360verin123
As we see this is not yet correct, as the candidates generate prior the mask. So this needs some more tweaking.
Masks you can define specific characters as well, for instance:
hashcat -m 9600 -a 3 doc_pass_hash.txt ?d?d?dhorse?d?d?d --username -o cracked_pass.txt --force
Session..........: hashcat
Status...........: Cracked
Hash.Type........: MS Office 2013
Hash.Target......: $office$*2013*100000*256*16*e77e386a8e68462d2a0a703...69fcf4
Time.Started.....: Sat May 30 17:24:32 2020 (28 secs)
Time.Estimated...: Sat May 30 17:25:00 2020 (0 secs)
Guess.Mask.......: ?d?d?dhorse?d?d?d [11]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........: 18 H/s (8.21ms) # Accel:128 Loops:32 Thr:1 Vec:8
Recovered........: 1/1 (100.00%) Digests, 1/1 (100.00%) Salts
Progress.........: 512/1000000 (0.05%)
Rejected.........: 0/512 (0.00%)
Restore.Point....: 0/100000 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:0-1 Iteration:99968-100000
Candidates.#1....: 123horse123 -> 112horse778
cat cracked_pass.txt
$office$*2013*100000*256*16*e77e386a8e68462d2a0a703718febbc9*08ee275ccf4946ae0e5922e9ff3114b7*0ab5fc00964f7ed4be9e45c77a33b441b2c4874d28e4bc30f38e99bfb169fcf4:123horse123
Cracked password in the end of the file: 123horse123
There is more to be read about rules and cracking with increased password lenght (--incremental) and combined attacks, but you get the idea.
Here are the official basic examples to get you started:
- [ Basic Examples ] -
Attack- | Hash- |
Mode | Type | Example command
==================+=======+==================================================================
Wordlist | $P$ | hashcat -a 0 -m 400 example400.hash example.dict
Wordlist + Rules | MD5 | hashcat -a 0 -m 0 example0.hash example.dict -r rules/best64.rule
Brute-Force | MD5 | hashcat -a 3 -m 0 example0.hash ?a?a?a?a?a?a
Combinator
| MD5 | hashcat -a 1 -m 0 example0.hash example.dict example.dict
Scenebuilder.exe will not start on my new PC either from IntelliJ, Windows explorer, or command line. It gives no errors and terminates instantly. On a hunch I tried running the included JVM in
C:\Program Files\SceneBuilder\runtime\bin
java --version
I get the log below and it shows an access exception that looks like a bug in the underlying JVM itself, not in the execution of the SceneBuilder code. I've been running JDK 11 on this AMD Ryzen just fine, and there are no DLLs sneaking in that do not belong there. I need help, thanks!
C:\Program Files\SceneBuilder\runtime\bin>java --version
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff8e3bf8f29, pid=12296, tid=11532
#
# JRE version: (11.0.1+13) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0.1+13-LTS, mixed mode, aot, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V [jvm.dll+0x1e8f29]
#
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\Tim\AppData\Local\Temp\\hs_err_pid12296.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
C:\Program Files\SceneBuilder\runtime\bin>type C:\Users\Tim\AppData\Local\Temp\\hs_err_pid12296.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff8e3bf8f29, pid=12296, tid=11532
#
# JRE version: (11.0.1+13) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0.1+13-LTS, mixed mode, aot, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V [jvm.dll+0x1e8f29]
#
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- S U M M A R Y ------------
Command Line:
Host: AMD Ryzen 5 2600 Six-Core Processor , 12 cores, 15G, Windows 10 , 64 bit Build 18362 (10.0.18362.778)
Time: Sat Apr 25 09:02:07 2020 Eastern Daylight Time elapsed time: 0 seconds (0d 0h 0m 0s)
--------------- T H R E A D ---------------
Current thread (0x000002312e8c6000): JavaThread "Unknown thread" [_thread_in_vm, id=11532, stack(0x0000005029100000,0x0000005029200000)]
Stack: [0x0000005029100000,0x0000005029200000], sp=0x00000050291feee0, free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1e8f29]
V [jvm.dll+0x1e73d3]
V [jvm.dll+0x37524b]
V [jvm.dll+0x7394a9]
V [jvm.dll+0x3e467d]
V [jvm.dll+0x3e6951]
C [java.exe+0x3603]
C [java.exe+0x178dc]
C [KERNEL32.DLL+0x17bd4]
C [ntdll.dll+0x6ce51]
siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), reading address 0x0000000000000000
Registers:
RAX=0x0000a780aacf8fcc, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0x0000000000000006
RSP=0x00000050291feee0, RBP=0x0000000000000000, RSI=0x000002312e8c6000, RDI=0x0000000000000000
R8 =0x0000000000000006, R9 =0x0000000000000006, R10=0x0000000000000006, R11=0x00000050291fe750
R12=0x0000000000000000, R13=0x00007ff8e3ff6d90, R14=0x000002312c602d98, R15=0x00000050291ff938
RIP=0x00007ff8e3bf8f29, EFLAGS=0x0000000000010256
Top of Stack: (sp=0x00000050291feee0)
0x00000050291feee0: 000002312c60db10 00007ff94e71c6c0
0x00000050291feef0: 00000050291ff938 000002312c602d98
0x00000050291fef00: 0000000000000000 0000000000000400
0x00000050291fef10: 00006181910f784b 00007ff8e3ff6d90
0x00000050291fef20: 0000000000000000 00000050291ff938
0x00000050291fef30: 000002312c602d98 00007ff8e3ff6d90
0x00000050291fef40: 0000000000000000 00007ff92e470000
0x00000050291fef50: 00007ff8e4282558 0000000000000000
0x00000050291fef60: 00007ff8e3bf7277 00007ff94e6d1b1c
0x00000050291fef70: 0000000000000000 00007ff8e40745dd
0x00000050291fef80: 0000a780aacf8fcc 000002312c602d98
0x00000050291fef90: 00007ff800000000 00007ff8e3ff6d90
0x00000050291fefa0: 000002312e8c6000 00000050291ff0c0
0x00000050291fefb0: 0000000000000000 00007ff8e3bf73d3
0x00000050291fefc0: 00007ff92e450000 00007ff92e47d088
0x00000050291fefd0: 00000050291ff0c0 000002312e8c6000
Instructions: (pc=0x00007ff8e3bf8f29)
0x00007ff8e3bf8f09: b8 00 00 00 48 8b 05 bc 40 85 00 48 33 c4 48 89
0x00007ff8e3bf8f19: 84 24 a0 00 00 00 48 8b e9 48 83 cb ff 48 ff c3
0x00007ff8e3bf8f29: 80 3c 0b 00 75 f7 33 f6 c6 44 24 30 01 40 38 35
0x00007ff8e3bf8f39: c2 62 88 00 74 17 40 38 35 a3 c9 87 00 75 0e 33
--------------- P R O C E S S ---------------
Threads class SMR info:
_java_thread_list=0x000002312c602d50, length=0, elements={
}
Java Threads: ( => current thread )
Other Threads:
[error occurred during error reporting (printing all threads), id 0xc0000005, EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff8e414e1fd]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
[error occurred during error reporting (printing compressed oops mode), id 0xc0000005, EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff8e4163b67]
Deoptimization events (0 events):
No events
Classes redefined (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
0x00007ff6af5a0000 - 0x00007ff6af5e7000 C:\Program Files\SceneBuilder\runtime\bin\java.exe
0x00007ff9512e0000 - 0x00007ff9514d0000 C:\WINDOWS\SYSTEM32\ntdll.dll
0x00007ff950230000 - 0x00007ff9502e2000 C:\WINDOWS\System32\KERNEL32.DLL
0x00007ff94e6c0000 - 0x00007ff94e963000 C:\WINDOWS\System32\KERNELBASE.dll
0x00007ff94fb00000 - 0x00007ff94fba3000 C:\WINDOWS\System32\ADVAPI32.dll
0x00007ff9502f0000 - 0x00007ff95038e000 C:\WINDOWS\System32\msvcrt.dll
0x00007ff951200000 - 0x00007ff951297000 C:\WINDOWS\System32\sechost.dll
0x00007ff94f9e0000 - 0x00007ff94fb00000 C:\WINDOWS\System32\RPCRT4.dll
0x00007ff94f390000 - 0x00007ff94f524000 C:\WINDOWS\System32\USER32.dll
0x00007ff94f2e0000 - 0x00007ff94f301000 C:\WINDOWS\System32\win32u.dll
0x00007ff94f530000 - 0x00007ff94f556000 C:\WINDOWS\System32\GDI32.dll
0x00007ff94e520000 - 0x00007ff94e6b4000 C:\WINDOWS\System32\gdi32full.dll
0x00007ff94f240000 - 0x00007ff94f2de000 C:\WINDOWS\System32\msvcp_win.dll
0x00007ff94e270000 - 0x00007ff94e36a000 C:\WINDOWS\System32\ucrtbase.dll
0x00007ff942260000 - 0x00007ff9424e5000 C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.18362.778_none_e6c6b761130d4fb8\COMCTL32.dll
0x00007ff947e60000 - 0x00007ff947e6a000 C:\WINDOWS\SYSTEM32\VERSION.dll
0x00007ff94f640000 - 0x00007ff94f976000 C:\WINDOWS\System32\combase.dll
0x00007ff94e370000 - 0x00007ff94e3f0000 C:\WINDOWS\System32\bcryptPrimitives.dll
0x00007ff950df0000 - 0x00007ff950e1e000 C:\WINDOWS\System32\IMM32.DLL
0x00007ff94aa90000 - 0x00007ff94aaa6000 C:\Program Files\SceneBuilder\runtime\bin\vcruntime140.dll
0x00007ff932a30000 - 0x00007ff932ada000 C:\Program Files\SceneBuilder\runtime\bin\msvcp140.dll
0x00007ff8e3a10000 - 0x00007ff8e4552000 C:\Program Files\SceneBuilder\runtime\bin\server\jvm.dll
0x00007ff950220000 - 0x00007ff950228000 C:\WINDOWS\System32\PSAPI.DLL
0x00007ff94ae30000 - 0x00007ff94ae54000 C:\WINDOWS\SYSTEM32\WINMM.dll
0x00007ff93c080000 - 0x00007ff93c089000 C:\WINDOWS\SYSTEM32\WSOCK32.dll
0x00007ff950d80000 - 0x00007ff950def000 C:\WINDOWS\System32\WS2_32.dll
0x00007ff94ae00000 - 0x00007ff94ae2d000 C:\WINDOWS\SYSTEM32\WINMMBASE.dll
0x00007ff94e4d0000 - 0x00007ff94e51a000 C:\WINDOWS\System32\cfgmgr32.dll
0x00007ff94e250000 - 0x00007ff94e261000 C:\WINDOWS\System32\kernel.appcore.dll
0x00007ff94a980000 - 0x00007ff94a991000 C:\Program Files\SceneBuilder\runtime\bin\verify.dll
0x00007ff93d1a0000 - 0x00007ff93d394000 C:\WINDOWS\SYSTEM32\DBGHELP.DLL
0x00007ff93cf30000 - 0x00007ff93cf5a000 C:\WINDOWS\SYSTEM32\dbgcore.DLL
0x00007ff92e470000 - 0x00007ff92e499000 C:\Program Files\SceneBuilder\runtime\bin\java.dll
0x00007ff92e450000 - 0x00007ff92e466000 C:\Program Files\SceneBuilder\runtime\bin\zip.dll
0x00007ff94aa80000 - 0x00007ff94aa8a000 C:\Program Files\SceneBuilder\runtime\bin\jimage.dll
dbghelp: loaded successfully - version: 4.0.5 - missing functions: none
symbol engine: initialized successfully - sym options: 0x614 - pdb path: .;C:\Program Files\SceneBuilder\runtime\bin;C:\WINDOWS\SYSTEM32;C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.18362.778_none_e6c6b761130d4fb8;C:\Program Files\SceneBuilder\runtime\bin\server
VM Arguments:
java_command: <unknown>
java_class_path (initial): <not set>
Launcher Type: SUN_STANDARD
[Global flags]
uint G1ConcRefinementThreads = 10 {product} {ergonomic}
uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
size_t InitialHeapSize = 267632960 {product} {ergonomic}
size_t MaxHeapSize = 4282127360 {product} {ergonomic}
uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
bool SegmentedCodeCache = true {product} {ergonomic}
bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
bool UseCompressedOops = true {lp64_product} {ergonomic}
bool UseG1GC = true {product} {ergonomic}
bool UseLargePagesIndividualAllocation = false {pd product} {ergonomic}
Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags
Environment Variables:
JAVA_HOME=C:\Users\Tim\jdk-13.0.2
PATH=...
USERNAME=Tim
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD
--------------- S Y S T E M ---------------
OS: Windows 10 , 64 bit Build 18362 (10.0.18362.778)
CPU:total 12 (initial active 12)
Memory: 4k page, system-wide physical 16335M (7401M free)
TotalPageFile size 38863M (AvailPageFile size 26823M)
current process WorkingSet (physical memory assigned to process): 8M, peak: 8M
current process commit charge ("private bytes"): 3M, peak: 3M
vm_info: Java HotSpot(TM) 64-Bit Server VM (11.0.1+13-LTS) for windows-amd64 JRE (11.0.1+13-LTS), built on Oct 6 2018 13:20:16 by "mach5one" with MS VC++ 15.5 (VS2017)
END.
I have two tables where Security holds the access bit mask for a given NTFS file system scan and FileSystemRights which equates to the string representations for the well known bit masks. I need to create a view which exposes the expected (not just proper) string representations for a given bit mask. The problem is several enum values composite and contain combinations of lower values, so the desired idea is not to repeat the implicit values.
For example, a value of 1179817 (Security.Id = 24) should only report ReadAndExecute and Synchronize, excluding ExecuteFile, ListDirectory, Read, ReadAttributes, ReadData, ReadExtendedAttributes, ReadPermissions and Traverse, as those are all part of ReadAndExecute (eg. ReadAndExecute & Read == Read). Its obviously correct to show them all, but a user wants only to see the non implicit values.
I'm lost within the constraints of SQL to produce a join that behaves like this without some abysmal nested case that would be a nightmare to look at.
Does a better programmatic approach exist?
FileSystemRights
================
Id Name Value
-- ---- -----
1 None 0
2 ListDirectory 1
3 ReadData 1
4 WriteData 2
5 CreateFiles 2
6 CreateDirectories 4
7 AppendData 4
8 ReadExtendedAttributes 8
9 WriteExtendedAttributes 16
10 ExecuteFile 32
11 Traverse 32
12 DeleteSubdirectoriesAndFiles 64
13 ReadAttributes 128
14 WriteAttributes 256
15 Write 278
16 Delete 65536
17 ReadPermissions 131072
18 Read 131209
19 ReadAndExecute 131241
20 Modify 197055
21 ChangePermissions 262144
22 TakeOwnership 524288
23 Synchronize 1048576
24 FullControl 2032127
25 GenericAll 268435456
26 GenericExecute 536870912
27 GenericWrite 1073741824
28 GenericRead 2147483648
Security
========
Id FileSystemRights IdentityReference
-- ---------------- -----------------
20 2032127 BUILTIN\Administrators
21 2032127 BUILTIN\Administrators
22 2032127 NT AUTHORITY\SYSTEM
23 268435456 CREATOR OWNER
24 1179817 BUILTIN\Users
25 4 BUILTIN\Users
26 2 BUILTIN\Users
MyView
======
SELECT s.Id AS SecurityId,
f.Name
FROM Security s
JOIN FileSystemRights f
ON CASE f.Value
WHEN 0 THEN s.FileSystemRights = f.Value
ELSE (s.FileSystemRights & f.Value) == f.Value
END
ORDER BY s.Id, f.Name;
Add the actual value of the name to the query.
Then wrap another query around that to filter out values for the same entry that are a subset of another value:
WITH AllValues(SecurityId, Name, Value) AS (
SELECT s.Id,
f.Name,
f.Value
FROM Security s
JOIN FileSystemRights f
ON CASE f.Value
WHEN 0 THEN s.FileSystemRights = f.Value
ELSE (s.FileSystemRights & f.Value) == f.Value
END
)
SELECT SecurityId,
Name
FROM AllValues
WHERE NOT EXISTS (SELECT *
FROM AllValues AS AV2
WHERE AV2.SecurityId = AllValues.SecurityId
AND (AV2.Value & AllValues.Value) != 0
AND AV2.Value > AllValues.Value
)
ORDER BY 1, 2;
This is an unresolved exam question of mine.
Can two Unix processes simultaneous write to different positions
in a single file?
Yes, the two processes will have their own file table entries
no, the shared i-node contains a single offset pointer
only one process will have write privilege
yes, but only if we operate using NFS
There is no file offset recorded in an inode so answer 2. is incorrect.
There is no documented reason for a process to have its access rights modified so 3. is incorrect.
NFS allows simultaneous access by processes on different hosts, the question here is for processes on the same host so NFS shouldn't make a difference.
Here is a shell script demonstrating the remaining answer 1. is correct:
# create a 10m file
dd if=/dev/zero of=/var/tmp/file bs=1024k count=10
# create two 1 MB files
cd /tmp
printf "aaaaaaaa" > aa
printf "bbbbbbbb" > bb
i=0
while [ $i -lt 17 ]; do
cat aa aa > aa.new && mv aa.new aa
cat bb bb > bb.new && mv bb.new bb
i=$((i+1))
done
ls -lG /var/tmp/file /tmp/aa /tmp/bb
# launch 10 processes that will write at different locations in the same file.
# Uses dd notrunc option for the file not to be truncated
# Uses GNU dd fdatasync option for unbuffered writes
i=0
while [ $i -lt 5 ]; do
(
dd if=/tmp/aa of=/var/tmp/file conv=notrunc,fdatasync bs=1024k count=1 seek=$((i*2)) 2>/dev/null &
dd if=/tmp/bb of=/var/tmp/file conv=notrunc,fdatasync bs=1024k count=1 seek=$((i*2+1)) 2>/dev/null &
) &
i=$((i+1))
done
# Check concurrency
printf "\n%d processes are currently writing to /var/tmp/file\n" "$(fuser /var/tmp/file 2>/dev/null | wc -w)"
# Wait for write completion and check file contents
wait
printf "/var/tmp/file contains:\n"
od -c /var/tmp/file
Its output shows ten processes successfully and simultaneously write to the very same file:
-rw-r--r-- 1 jlliagre 1048576 oct. 30 08:25 /tmp/aa
-rw-r--r-- 1 jlliagre 1048576 oct. 30 08:25 /tmp/bb
-rw-r--r-- 1 jlliagre 10485760 oct. 30 08:25 /var/tmp/file
10 processes are currently writing to /var/tmp/file
/var/tmp/file contains:
0000000 a a a a a a a a a a a a a a a a
*
4000000 b b b b b b b b b b b b b b b b
*
10000000 a a a a a a a a a a a a a a a a
*
14000000 b b b b b b b b b b b b b b b b
*
20000000 a a a a a a a a a a a a a a a a
*
24000000 b b b b b b b b b b b b b b b b
*
30000000 a a a a a a a a a a a a a a a a
*
34000000 b b b b b b b b b b b b b b b b
*
40000000 a a a a a a a a a a a a a a a a
*
44000000 b b b b b b b b b b b b b b b b
*
50000000
Definition:
Yes, the two processes will have their own file table entries.
If the file is opened twice with the open function the two file descriptor is created.
Each file descriptor have seperate file status flags.
So the two file descriptor have a write permission file descriptor1 and file descriptor2 have initial position of point to first character to file.
If we specify some position to both descriptor and write in file it can be tested easily.
The content of file.txt
My name is Chandru. This is an empty file.
Coding for testing:
#include<stdio.h>
#include<fcntl.h>
#include<stdlib.h>
main()
{
int fd1, fd2;
if((fd1=open("file.txt", O_WRONLY)) <0){
perror("Error");
exit(0);
}
if((fd2=open("file.txt", O_WRONLY)) < 0) {
perror("Error");
exit(0);
}
if(lseek(fd1,20,SEEK_SET) != 20)
{
printf("Cannot seek\n");
exit(0);
}
if(write(fd1,"testing",7) != 7)
{
printf("Error write\n");
exit(0);
}
if(lseek(fd2,10,SEEK_SET) != 10)
{
printf("Cannot seek\n");
exit(0);
}
if(write(fd2,"condition",9) != 9)
{
printf("Error write\n");
exit(0);
}
}
Output:
After that my output is
My name isconditionitesting empty file.
Yes, they can of course, with the following caveats:
Depending on open() mode, one process can easily wipe the file contents
Depending on scheduling, the order of write operations is not deterministic
There is no mandatory locking (in general) - careful design calls for advisory locking.
If they write in the same area using buffered I/O results can be non-deterministic.
Lets say you want 5 threads to process data simultaneous. Also assume, you have 89 tasks to process.
Off the bat you know 89 / 5 = 17 with a remainder of 4. The best way to split up tasks would be to have 4 (the remainder) threads process 18 (17+1) tasks each and then have 1 (# threads - remainder) thread to process 17.
This will eliminate the remainder. Just to verify:
Thread 1: Tasks 1-18 (18 tasks)
Thread 2: Tasks 19-36 (18 tasks)
Thread 3: Tasks 37-54 (18 tasks)
Thread 4: Tasks 55-72 (18 tasks)
Thread 5: Tasks 73-89 (17 tasks)
Giving you a total of 89 tasks completed.
I need a way of getting the start and ending range of each thread mathematically/programmability; where the following should print the exact thing I have listed above:
$NumTasks = 89
$NumThreads = 5
$Remainder = $NumTasks % $NumThreads
$DefaultNumTasksAssigned = floor($NumTasks / $NumThreads)
For $i = 1 To $NumThreads
if $i <= $Remainder Then
$NumTasksAssigned = $DefaultNumTasksAssigned + 1
else
$NumTasksAssigned = $DefaultNumTasksAssigned
endif
$Start = ??????????
$End = ??????????
print Thread $i: Tasks $Start-$End ($NumTasksAssigned tasks)
Next
This should also work for any number of $NumTasks.
Note: Please stick to answering the math at hand and avoid suggesting or assuming the situation.
Why? Rather then predetermining the scheduling order, stick all of the tasks on a queue, and then have each thread pull them off one by one when they're ready. Then your tasks will basically run "as fast as possible".
If you pre-allocated, then one thread may be doing a particularly long bit of processing and blocking the running of all the tasks stuck behind it. Using the queue, as each task finishes and a thread frees up, it grabs the next task and keeps going.
Think of it like a bank with 1 line per teller vs one line and a lot of tellers. In the former, you might get stuck behind the person depositing coins and counting it out one by one, the latter you get to the next available teller, while Mr. PocketChange counts away.
I second Will Hartung 's remark. You may just feed them one task at a time (or a few tasks at a at a time, depending if there's much overhead, i.e. if individual tasks get typically completed very fast, relative to the cost of starting/recycling threads). Your subsequent comments effectively explain that your "threads" carry a heavy creation cost, and hence your desire to feed them once with as much work as possible, rather than wasting time creating new "thread" each fed a small amount of work.
Anyway... going to the math question...
If you'd like to assign tasks just once, the following formula, plugged in lieu of the the ????????? in your logic, should do the trick:
$Start = 1
+ (($i -1) * ($DefaultNumTasksAssigned + 1)
- (floor($i / ($Remainder + 1)) * ($i - $Remainder))
$End = $Start + $NumTasksAssigned -1
The formula is explained as follow:
1 is for the fact that your display / logic is one-based not zero-based
The second term is because we generally add ($DefaultNumTasksAssigned + 1) with each iteration.
The third term provides a correction for the last few iterations.
Its first part, (floor($i / ($Remainder + 1)) provides 0 until $i reaches the first thread
that doesn't receive one extra task, and 1 thereafter.
The second part express by how much we need to correct.
The formula for $End is easier, the only trick is the minus 1, it is because the Start and End values are inclusive (so for example between 1 an 19 there are 19 tasks not 18)
The following slightly modified piece of logic should also work, it avoids the "fancy" formula by keeping a running tab of the $Start variable, rather than recomputing it each time..
$NumTasks = 89
$NumThreads = 5
$Remainder = $NumTasks % $NumThreads
$DefaultNumTasksAssigned = floor($NumTasks / $NumThreads)
$Start = 1
For $i = 1 To $NumThreads
if $i <= $Remainder Then // fixed here! need <= because $i is one-based
$NumTasksAssigned = $DefaultNumTasksAssigned + 1
else
$NumTasksAssigned = $DefaultNumTasksAssigned
endif
$End = $Start + $NumTasksAssigned -1
print Thread $i: Tasks $Start-$End ($NumTasksAssigned tasks)
$Start = $Start + $NumTasksAssigned
Next
Here's a Python transcription of the above
>>> def ShowWorkAllocation(NumTasks, NumThreads):
... Remainder = NumTasks % NumThreads
... DefaultNumTasksAssigned = math.floor(NumTasks / NumThreads)
... Start = 1
... for i in range(1, NumThreads + 1):
... if i <= Remainder:
... NumTasksAssigned = DefaultNumTasksAssigned + 1
... else:
... NumTasksAssigned = DefaultNumTasksAssigned
... End = Start + NumTasksAssigned - 1
... print("Thread ", i, ": Tasks ", Start, "-", End, "(", NumTasksAssigned,")")
... Start = Start + NumTasksAssigned
...
>>>
>>> ShowWorkAllocation(89, 5)
Thread 1 : Tasks 1 - 18 ( 18 )
Thread 2 : Tasks 19 - 36 ( 18 )
Thread 3 : Tasks 37 - 54 ( 18 )
Thread 4 : Tasks 55 - 72 ( 18 )
Thread 5 : Tasks 73 - 89 ( 17 )
>>> ShowWorkAllocation(11, 5)
Thread 1 : Tasks 1 - 3 ( 3 )
Thread 2 : Tasks 4 - 5 ( 2 )
Thread 3 : Tasks 6 - 7 ( 2 )
Thread 4 : Tasks 8 - 9 ( 2 )
Thread 5 : Tasks 10 - 11 ( 2 )
>>>
>>> ShowWorkAllocation(89, 11)
Thread 1 : Tasks 1 - 9 ( 9 )
Thread 2 : Tasks 10 - 17 ( 8 )
Thread 3 : Tasks 18 - 25 ( 8 )
Thread 4 : Tasks 26 - 33 ( 8 )
Thread 5 : Tasks 34 - 41 ( 8 )
Thread 6 : Tasks 42 - 49 ( 8 )
Thread 7 : Tasks 50 - 57 ( 8 )
Thread 8 : Tasks 58 - 65 ( 8 )
Thread 9 : Tasks 66 - 73 ( 8 )
Thread 10 : Tasks 74 - 81 ( 8 )
Thread 11 : Tasks 82 - 89 ( 8 )
>>>
I think you've solved the wrong half of your problem.
It's going to be virtually impossible to precisely determine the time it will take to complete all your tasks, unless all of the following are true:
your tasks are 100% CPU-bound: that is, they use 100% CPU while running and don't need to do any I/O
none of your tasks have to synchronize with any of your other tasks in any way
you have exactly as many threads as you have CPUs
the computer that is running these tasks is not performing any other interesting tasks at the same time
In practice, most of the time, your tasks are I/O-bound rather than CPU-bound: that is, you are waiting for some external resource such as reading from a file, fetching from a database, or communicating with a remote computer. In that case, you only make things worse by adding more threads, because they're all contending for the same scarce resource.
Finally, unless you have some really weird hardware, it's unlikely you can actually have exactly five threads running simultaneously. (Usually processor configurations come in multiples of at least two.) Usually the sweet spot is at about 1 thread per CPU if your tasks are very CPU-bound, about 2 threads per CPU if the tasks spend half their time being CPU-bound and half their time doing IO, etc.
tl;dr: We need to know a lot more about what your tasks and hardware look like before we can advise you on this question.