Add a flatten argument to ssh.libs

[ValekoZ]

This option makes us able to avoid getting the file tree of the remote server and just downloads the desired files in the output folder:

>>> shell = ssh(host='example.org')
>>> shell.libs('/bin/sh', directory = 'libs', flatten = True)
{'/path/to/cwd/libs/libc.so.6': 22708292161536, '/path/to/cwd/libs/ld-linux-x86-64.so.2': 22708294139904, '/path/to/cwd/libs/dash': 0}

[peace-maker]

We should emit a warning on duplicate files when the filename is equal but the remote path was different.

[ValekoZ]

Do you think it should just emit a warning and skip the download ? Or should it renames the file to something like {filename}_duplicate and then {filename}_duplicate2, etc. ?

[peace-maker]

I guess renaming them is better instead of discarding used libraries, but I'd switch to the unflattened structure anyways if this happens. We should log a warning in any case. I don't know how stable the order of the loaded libraries is in ldd and if you can be sure you always want the first matching library on every run.

[peace-maker]

Can you document that fallback behaviour in the argument description please?

[ValekoZ]

Yep, I just did it, sorry I forgot that

[peace-maker]

Cool, it would be awesome if you could add tests for the function testing withflatten on and off and with and without duplicates.

We're using doctests, check out other examples in the function docs comments of other functions in the file!

[ValekoZ]

I just tried to create unit tests for this functionnality, but I was wondering what design did you like better ?
I have the choice between creating minimal ELF files dynamically using lief for creating a binary with 2 libs with the same filename, or just dropping the binaries I created in the repo so that I don't have to add a dependency for the tests

[peace-maker]

I think adding another small binary to the pwnlib/data/elf folder seems fair. It doesn't have to be runnable I guess, so referencing duplicate not-existing libraries might be fine.

[ValekoZ]

Can you check that I added the tests correctly ? It says all tests passed but I'm not even sure the tests were run .. 😅

[peace-maker]

I think it's saner to abort on duplicate libraries instead of silently falling back to unflattened output. The locations would be different than what the script expects and it would break anyways.

but I'd switch to the unflattened structure anyways if this happens

I was talking about me using this feature, not how I'd imagine it to behave automatically. Sorry for the confusion.

You can test if the tests are run by adding something obviously broken and see if it breaks.

[peace-maker]

Nice, the CI ran now and revealed some problems with your tests. Can you strip the binaries to reduce size please?

[peace-maker]

Please create an __init__.py file and use pwnlib.data.elf.ssh_libs.get("duplicate") etc. below. Check the the __init__.py in one of the other sub folders. The cwd has to be writeable and it apparently isn't in CI, when you change to the pwnlib data directory.

[ValekoZ]

Apparently, the elf I pushed are corrupted (I can't run ldd on them), I'll try to fix that ^^

[ValekoZ]

Sorry for all the commits, I can't run the tests on my computer (it doesn't work, I don't know why)
I still have an issue and I don't understand why, I stop for today I'll check that later ^^

[peace-maker]

@Arusekk can you have a look please?

[peace-maker]

Bleh, I've tried to get the tests to work nicely but don't want to wrestle with -rpath $ORIGIN and friends anymore. I'll add this now and deal with the test again later.