state presented at PHDays VI, 17 May 2016, with slides

Author: AlekseyCherepanov

README.md

@@ -2,43 +2,18 @@
 
 john-devkit is an advanced code generator for [John the Ripper](https://github.com/magnumripper/JohnTheRipper). It aims to separate crypto primitives (sha-512, md5, crypt, pbdkf2 and so on), optimizations (interleave, loop unrolling, round reversing, bitslice and so on) and output for different computing devices (cpu, sse, gpu, fpga).
 
-john-devkit uses its own domain specific language (dsl) on top of Python to describe crypto algorithms in an abstract way. Also there is a kind of instruction language for intermediate representation (referenced as "bytecode"). So there are two levels of code generation: dsl -> bytecode -> platform's language (for instance C for cpu).
+john-devkit uses its own domain specific language (dsl) on top of Python to describe crypto algorithms in an abstract way. Also there is a kind of instruction language for intermediate representation (referenced as "bytecode"). So there are two levels of code generation: dsl -> intermediate representation -> platform's language (for instance C for cpu).
 
-"bytecode" is a wrong word for the intermediate language/representation and will be fixed soon.
+"bytecode" is a wrong word for the intermediate language/representation and will be fixed in code someday.
 
 ## Current State
 
 The current implementation is a Proof of Concept.
 
-There is no documentation for dsl and bytecode. It is not obvious how many times everything will be changed drastically.
+There is no documentation for dsl and intermediate. It is not obvious how many times everything will be changed drastically.
 
 There is a draft of hash parsing library by Alexander Cherepanov. It is not finished (and works by pure luck). It is included into john-devkit but it will be removed when the library become a persistent part of John the Ripper.
 
-7 "raw" formats were implemented: raw-sha256, raw-sha224, raw-sha512, raw-sha384, raw-md4, raw-md5, raw-sha1. SHA-2 family got speed up (~12% for SHA-256, ~5% for SHA-512). md5, md4 and sha1 got noticable slowdown (it needs investigation). (Such speed up for raw-sha256 may be caused by changes in test vector. Accurate benchmarks are needed.)
-
-raw-sha256 lost cisco hashes (base64 encoded) so benchmarks differ from John the Ripper not only due to optimizations.
-
-The current main priority is to try more complex formats like sha256crypt.
-
-john-devkit has now:
-  * formats: raw-sha256, raw-sha224, raw-sha512, raw-sha384, raw-md4, raw-md5, raw-sha1
-  * optimizations: vectorization (sse only), early reject, interleave, loop unrolling, additional batching in crypt_all() method
-
-broken:
-  * reverse (there is only initial implementation, it has problems with vectorization)
-  * bitslice (there is no support in template file; no long vectors, only regular ints for vectors)
-  * output to standalone program (it will not be fixed soon)
-
-Nearest plans:
-  * output to avx and avx2 for better benchmarking
-  * fix bitslice and play with it
-  * investigate md5/md4 slowdown (on sse)
-  * fix reverse
-  * sha2crypt scheme
-  * output formats for john on gpu
-  * incorporate on-gpu mask-mode
-  * try kernel splitting
-
 ## Usage
 
 To run john-devkit, you need
@@ -54,13 +29,21 @@ Then the following will work:
 
 After the first run (for each format), you need to cd into JohnTheRipper/src/ , rerun ./configure script (otherwise you'll get "Unknown ciphertext format name requested") and rerun the command.
 
+Calling `format_abstract_all.py` can give various formats. It is tricky though because it does not work with `run_format_raw.sh` and may require changes to enable or disable certain formats.
+
 ## Files layout
 
 `algo_*.py` are abstract definitions of crypto primitives written in DSL.
 
 `bytecode_main.py` is the main library to transform intermediate representation.
 
-`format_john_*.py` are config files to output certain formats for john.
+`bytecode_*.py` are other files with filters for intermediate representation.
+
+`format_abstract_all.py` is a big mess with code to generate and test ~100 formats for john. TODO: split the file.
+
+`format_john_*.py` are other config files to output certain formats for john.
+
+`format_*.py` are config files to produce other output files (1 hash algo per file usually).
 
 `lang_main.py` and `lang_spec.py` contain support code for DSL.
 
@@ -74,18 +57,19 @@ After the first run (for each format), you need to cd into JohnTheRipper/src/ ,
 
 `run_format_raw.sh` is a script to call `format_john_*.py` quickly for "raw" formats, call disassembler and count instructions/size of crypt_all() method.
 
-`slides_2015-05-26_phdays_v.src.org` is a textual source of slides for PHDays V. See below.
+`slides_*.src.org` are textual source files of slides for talks. See below.
 
 `t_raw.c` is a C template for "raw" formats. It contains padding and byteswap for endianity fix. It is derived from code of John the Ripper. See below about its license.
 
-`util_ui.py` is a miscellaneous library.
+`t_*` are other templates.
 
-## Slides for PHDays V
+`util_*.py` are miscellaneous library files.
+
+## Slides for PHDays V, 2015
 
 Benchmarks demonstrated at PHDays V are quite inaccurate: raw-sha256 and raw-sha224 are more likely to be only 12% over John the Ripper's speed.
 
-Source file of slides for talk about john-devkit at PHDays V is in
-`slides_2015-05-26_phdays_v.src.org`
+Source file of slides for talk about john-devkit at PHDays V is in `slides_2015-05-26_phdays_v.src.org`.
 
 The slides contain code example from Keccak (from [JohnTheRipper/src/KeccakF-1600-unrolling.macros](https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/src/KeccakF-1600-unrolling.macros)) and code example from NetBSD's libcrypt (from [here](https://github.com/rumpkernel/netbsd-userspace-src/blob/3280867f12bbd346f39d5a4efb41fcf9b087bf33/lib/libcrypt/hmac_sha1.c)). Everything else is under the following license:
 
@@ -97,9 +81,15 @@ Code examples are between `>>>>` and `<<<<` . `#` in text is quoted with `\`: so
 
 TODO: a link to .pdf file, a script to compile slides.
 
+## Slides for PHDays VI, 2016
+
+Source file of slides for talk about john-devkit at PHDays VI is in `slides_2016-05-17_phdays_vi.src.org`. The code example is in `simple_example.py`.
+
+TODO: a link to .pdf file.
+
 ## Usage without John the Ripper
 
-The idea to write a hash algo in Python and get optimized C code is very attractive. But there are limitations with john-devkit: john-devkit is not suitable for regular applications. It is a really bad idea to use john-devkit in most cases.
+The idea to write a hash algo in Python and get optimized C code is very attractive. But there are limitations with john-devkit: john-devkit is not suitable for regular applications. It is a really bad idea to use john-devkit in most cases. Please use standard/good libraries for hashing; for instance Argon2 (see [Password Hashing Competitions](https://password-hashing.net/)), [yescrypt](http://www.openwall.com/yescrypt/) or [phpass for php](http://www.openwall.com/phpass/).
 
 It is possible to separate john-devkit and John the Ripper (use custom C template, fix output in `output_c.py` for some instructions that depend on `pseudo_intrinsics.h` and/or `johnswap.h` or pull the headers into your application if the licenses permit).
 
@@ -109,8 +99,6 @@ Also it should be easier to implement and optimize 1 hash algo manually than usi
 
 There is a totally "no-no" thing for regular applications: john-devkit does not care about security, there are no defensive tricks, for instance john-devkit does not prevent information disclose through timings, so produced code is weak against a range of attacks.
 
-Please use standard/good libraries for hashing. For instance phpass for php.
-
 ## License
 
 Currently, files generated by john-devkit are subject for original license of John the Ripper. See below.

algo_aes_decrypt.py

@@ -0,0 +1,28 @@
+# AES decryption, abstract
+
+# Copyright © 2016 Aleksey Cherepanov <[email protected]>
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted.
+
+Var.setup('be', 4)
+
+msg = input_salt()
+key = input_key()
+
+assume_length(msg, 16, 16)
+assume_length(key, 16, 16)
+
+include('aes')
+
+r = AES(key).decrypt(msg)
+
+# We've got 16 ints with 1 bytes. Let's pack them.
+
+# print_many(*r)
+
+o = []
+for i in range(0, 16, 4):
+    o.append((r[i + 0] << 24) ^ (r[i + 1] << 16) ^ (r[i + 2] << 8) ^ r[i + 3])
+
+s = bytes_join_nums(*o)
+output_bytes(s)

algo_aes_encrypt.py

@@ -0,0 +1,28 @@
+# AES encryption, abstract
+
+# Copyright © 2016 Aleksey Cherepanov <[email protected]>
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted.
+
+Var.setup('be', 4)
+
+msg = input_salt()
+key = input_key()
+
+assume_length(msg, 16, 16)
+assume_length(key, 16, 16)
+
+include('aes')
+
+r = AES(key).encrypt(msg)
+
+# We've got 16 ints with 1 bytes. Let's pack them.
+
+# print_many(*r)
+
+o = []
+for i in range(0, 16, 4):
+    o.append((r[i + 0] << 24) ^ (r[i + 1] << 16) ^ (r[i + 2] << 8) ^ r[i + 3])
+
+s = bytes_join_nums(*o)
+output_bytes(s)

algo_crypt_sha2.py

@@ -0,0 +1,210 @@
+# crypt with sha2 scheme
+# %% draft implementation
+# %% sha512 only
+
+# Copyright © 2016 Aleksey Cherepanov <[email protected]>
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted.
+
+# http://www.akkadia.org/drepper/SHA-crypt.txt
+
+# Var.setup('be', args['size'])
+
+sha = load_hfun('sha512')
+
+# Algo
+
+key = input_key()
+salt = input_salt()
+rounds = input_rounds()
+
+key_length = bytes_length(key)
+salt_length = bytes_length(salt)
+# key_bit_length = get_bit_length(key)
+
+ks = bytes_concat(key, salt)
+ksk = bytes_concat(ks, key)
+b = invoke_hfun(sha, ksk)
+
+a = sha_init()
+ksa = bytes_concat(ks, b)
+
+# 11
+# maximal password length is 125
+# %% хорошо бы тут использовать другой тип
+c = new_var()
+c // key_length
+bb = new_bytes()
+bytes_assign(bb, ksa)
+cycle_while_begin('step11')
+cycle_while('step11', c > 0)
+
+if_condition('c1', c & 1)
+bytes_assign(bb, bytes_concat(bb, b))
+if_else('c1')
+bytes_assign(bb, bytes_concat(bb, key))
+if_end('c1')
+
+c // (c >> 1)
+cycle_end('step11')
+
+# 12
+a = invoke_hfun(sha, bb)
+
+# %% not inclusive
+kkk = new_bytes()
+unused = cycle_range('step14', 0, key_length - 1, 1)
+bytes_assign(kkk, bytes_concat(kkk, key))
+cycle_end('step14')
+dp = invoke_hfun(sha, kkk)
+
+
+# %%% я остановился тут
+
+p = fill_string(key_length, dp_string, dp_string_length)
+p_length = key_length
+
+# 18
+ds = sha_init()
+# for i in range(16 + a[0]):
+#     sha_update(ds, salt)
+# %% big endian?
+# %% not inclusive
+c = cycle_range('setup_s', 0, new_const(16) + ((a[0] & (0xff << 56)) >> 56) - 1, 1)
+sha_update(ds, salt, salt_length)
+cycle_end('setup_s')
+ds = sha_final(ds)
+
+ds_string = digest_to_string(ds)
+ds_string_length = 8 * 8
+
+s = fill_string(salt_length, ds_string, ds_string_length)
+s_length = salt_length
+
+# 21
+
+# # pseudo code
+# for r in range(rounds - 1):
+#     c = sha_init()
+#     sha_update(c, p if r & 1 else ac)
+#     if r % 3 != 0: sha_update(c, s)
+#     if r % 7 != 0: sha_update(c, p)
+#     sha_update(c, ac if r & 1 else p)
+#     c = sha_final(c)
+#     ac = c
+
+# Notice that computation of some blocks maybe lifted from the loop
+
+# Statistics of sequences for first 5k rounds:
+# perl -e 'for (0 .. 5000) { if ($_ & 1) { print " ac," } else { print " p," } print " s," if $_ % 3; print " p," if $_ % 7; if ($_ & 1) { print " p" } else { print " ac" } print "\n" }' | sort | uniq -c
+#     119  ac, p
+#     714  ac, p, p
+#     238  ac, s, p
+#    1429  ac, s, p, p
+#     120  p, ac
+#     714  p, p, ac
+#     238  p, s, ac
+#    1429  p, s, p, ac
+#   =5001, 1667 ps and psp may be brought upper
+
+# The full cycle is 2 * 3 * 7 == 42 rounds, having such unroll it is
+# possible to avoid most "if"s in the loop. There are 21 variants
+# Without ac/p.
+# %% Right?
+
+# %% It'd be nice to do such optimization automatically...
+
+# # %% add 0x80 and compute lengths
+# # Only ac is variable. Other parts may be extracted:
+# pp = string_concat(p, p)
+# sp = string_concat(s, p)
+# spp = string_concat(s, p, p)
+# ps = string_concat(p, s)
+# psp = string_concat(p, s, p)
+# pp = string_to_ints(pp)
+# sp = string_to_ints(sp)
+# spp = string_to_ints(spp)
+# ps = string_to_ints(ps)
+# psp = string_to_ints(psp)
+
+# %% It is possible to reduce memory footprint storing pp in spp, and
+# sp in psp; also it is possible to store ps inside psp, but it may
+# be needed to pass lengths then
+
+# for r in range(rounds - 1):
+#     c = sha_init()
+#     if r & 1:
+#         if r % 3 and r % 7:
+#             sha_update(c, psp)
+#         elif r % 7:
+#             sha_update(c, pp)
+#         elif r % 3:
+#             sha_update(c, ps)
+#         else:
+#             sha_update(c, p)
+#         sha_update(c, ac)
+#     else:
+#         sha_update(c, ac)
+#         if r % 3 and r % 7:
+#             sha_update(c, spp)
+#         elif r % 7:
+#             sha_update(c, pp)
+#         elif r % 3:
+#             sha_update(c, sp)
+#         else:
+#             sha_update(c, p)
+#     c = sha_final(c)
+#     ac = c
+
+# r = cycle_range('main', 0, rounds, 1)
+# # %% Compute 8?
+# c_array = [Var() for i in range(8)]
+# # ints_concat is "macro", free
+# f = lambda x, y: sha_ints(ints_concat(x, y))
+# c = iff(r & 1,
+#         # %% lift computation of some blocks from the cycle
+#         iff(r % 3 and r % 7, f(psp, c),
+#             iff(r % 7, f(pp, c),
+#                 iff(r % 3, f(ps, c),
+#                     f(p, c)))),
+#         iff(r % 3 and r % 7, f(c, spp),
+#             iff(r % 7, f(c, pp),
+#                 iff(r % 3, f(c, sp),
+#                     f(c, p)))))
+# for i, v in enumerate(c):
+#     c_array[i] // c[i]
+# cycle_end('main')
+
+ac = a
+
+# %% not inclusive
+r = cycle_range('main', 0, rounds - 1, 1)
+c = sha_init()
+# print_var(r)
+# print_digest(ac)
+ac_string = digest_to_string(ac)
+ac_string_length = 8 * 8
+if_condition('r1', r & 1)
+sha_update(c, p, p_length)
+if_else('r1')
+sha_update(c, ac_string, ac_string_length)
+if_end('r1')
+if_condition('r3', r % 3)
+sha_update(c, s, s_length)
+if_end('r3')
+if_condition('r7', r % 7)
+sha_update(c, p, p_length)
+if_end('r7')
+if_condition('r2', r & 1)
+sha_update(c, ac_string, ac_string_length)
+if_else('r2')
+sha_update(c, p, p_length)
+if_end('r2')
+# мы затираем 'a'
+# %% memory leak?
+ac // sha_final(c)
+cycle_end('main')
+
+# %% avoid hardcoded value, how?
+for i in range(8):
+    output(ac[i])