How can I simulate hidden service (onion service) using Shadow?

[Allgot]

Hi! First of all, we have been studying this wonderful simulator for a while and would like to thank for your hard work in maintaining this project.

My team is working to evaluate the effectiveness of some attacks against hidden services, using Shadow. We could find some hints from this old archive, but it is out-dated and not compatible with the current version of Shadow.

It seems that we can build our configuration file from the ground up by referring to the old archive. However, before doing so, we want to make sure that this is not reinventing the wheel. Is there any updated, publicly available configuration file (i.e., shadow.yaml file)?

Thank you :)

[Allgot]

By the way, we have checked several artifacts from recent studies (e.g., explainwf), but we couldn't figure out how to configure the hidden service, unfortunately :(

[robgjansen]

Yes, you should use tornettools to help you generate new Tor network configurations that can run in shadow. There are several steps involved as explained in the README of that project. You can use the onion options during the generate step to configure hidden services: see tornettools generate --help for more details.

[sporksmith]

With PoW enabled, and with the high intro point turn-over, I had quite a few stream-errors in the first 8 sim minutes. With the above patch to disable high intro point turn-over, I had none in the first 8 minutes.

Didn't have a chance yet to do this comparison without PoW enabled, so still not sure this particular issue is worse in the PoW case or if there's something else going on.

[Allgot]

Wow, I appreciate your thorough investigation. Let me outline the entire experimentation process so that you can easily and precisely replicate our conditions. (I have manually verified that this accurately reproduces the problem!)

# version information
# shadow: commit (main) 72bd35f81b551b6f54d278d8e97a6c55648d371e
# tor: commit (main) 0cccc7223c52512b5e0f47819f1a02740e5dad9b
# !!You don't need to modify tor!!

# Staging
tornettools stage consensuses-2023-04 server-descriptors-2023-04 userstats-relay-country.csv tmodel-ccs2018.github.io --onionperf_data_path onionperf-2023-04 --bandwidth_data_path bandwidth-2023-04.csv --geoip_path tor/src/config/geoip

# Generation
tornettools generate relayinfo_staging_2023-04-01--2023-04-30.json userinfo_staging_2023-04-01--2023-04-30.json networkinfo_staging.gml tmodel-ccs2018.github.io/ --network_scale 0.01 --process_scale 0.01 --torperf_num_exit 0 --torperf_num_onion_service 0 --onion_service_user_scale 0.5 --exit_user_scale 0.5 --prefix repro

# Simulation
tornettools simulate -a "--model-unblocked-syscall-latency=false --parallelism=48 --seed=666 --template-directory=shadow.data.template" repro

# Save results
mv repro/shadow.data repro/pow_off
mv repro/shadow.log repro/pow_off.log

# Turn PoW on
echo "HiddenServicePoWDefensesEnabled 1" >> repro/conf/tor.onionservice.torrc

# Simulation
tornettools simulate -a "--model-unblocked-syscall-latency=false --parallelism=48 --seed=666 --template-directory=shadow.data.template" repro

# Save results
mv repro/shadow.data repro/pow_on
mv repro/shadow.log repro/pow_on.log

---

Following your suggested patch, we conducted the same experiments as before. Attached are two images displaying the results, and it seems that the problem has been solved. Thanks! (However, I still don't know why there are always some failing clients.)

---

I'm suspicious that this will cause problems in a tornettools-generated simulation, because it uses a small number of servers and clients to simulate a larger number (see e.g. --process_scale).

Then can we conclude that increasing the process_scale towards 1 is the correct approach for conducting accurate experiments?

Once more, thank you for your detailed investigation.

[sporksmith]

Then can we conclude that increasing the process_scale towards 1 is the correct approach for conducting accurate experiments?

It depends exactly what you're trying to accomplish. If you just don't want the intro-point-dos-defense to trigger, you can just not enable it (the tornettools-generated configs shouldn't enable it by default).

If you are trying to answer questions about what behavior might be like on the real network in the face of some particular load or DoS, then you'll need to think carefully about how to model that. tornettools generate --help includes many options that may be relevant, including --process_scale, --server_scale, and --load_scale

[sporksmith]

Related: https://gitlab.torproject.org/tpo/core/tor/-/issues/40922

this issue is now fixed at tor HEAD. https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/802

[Allgot]

Related: https://gitlab.torproject.org/tpo/core/tor/-/issues/40922

this issue is now fixed at tor HEAD. https://gitlab.torproject.org/tpo/core/tor/-/merge_requests/802

I'm glad to hear that our report was helpful in addressing the bug! I believe we can now properly set up our experiments. Thank you :)