From 4135a6bfa135f812e737b8cdffcbdfac3d05db81 Mon Sep 17 00:00:00 2001 From: FiveMovesAhead Date: Tue, 6 May 2025 06:48:27 +0100 Subject: [PATCH] Update documentation for native runtime. --- README.md | 7 +- docs/guides/innovating.md | 193 ++++++++++++++++++++---------- tig-algorithms/README.md | 14 +-- tig-benchmarker/README.md | 53 ++------ tig-benchmarker/common/structs.py | 1 + tig-binary/README.md | 39 ++++++ tig-runtime/README.md | 97 +++++++++++++++ tig-runtime/src/main.rs | 2 +- tig-verifier/README.md | 89 ++++++++++++++ tig-worker/README.md | 156 ++++++++++++------------ 10 files changed, 458 insertions(+), 193 deletions(-) create mode 100644 tig-binary/README.md create mode 100644 tig-runtime/README.md create mode 100644 tig-verifier/README.md diff --git a/README.md b/README.md index a60db23..606b6db 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,6 @@ This repository contains the implementation of The Innovation Game (TIG). * [TIG Documentation](https://docs.tig.foundation/) * [TIG Whitepaper](docs/whitepaper.pdf) -* [TIG Tech Explainer](docs/tech/1_basics.md) * [TIG Licensing Explainer](docs/guides/anatomy.md) * [Getting Started with Innovating](docs/guides/innovating.md) * [Implementations vs Breakthroughs](docs/guides/breakthroughs.md) @@ -16,14 +15,16 @@ This repository contains the implementation of The Innovation Game (TIG). * [tig-algorithms](./tig-algorithms/README.md) - A Rust crate that hosts algorithm submissions made by Innovators in TIG * [tig-benchmarker](./tig-benchmarker/README.md) - Python scripts for running TIG's benchmarker in master/slave configuration +* [tig-binary](./tig-binary/README.md) - A Rust crate that wraps an algorithm from [`tig-algorithm`](./tig-algorithms/README.md) for compilation into a shared object. * [tig-breakthroughs](./tig-breakthroughs/README.md) - A folder that hosts submissions of algorithmic methods made by Innovators in TIG. * [tig-challenges](./tig-challenges/README.md) - A Rust crate that contains the implementation of TIG's challenges (computational problems adapted for proof-of-work) * [tig-protocol](./tig-protocol/README.md) - A Rust crate that contains the implementation of TIG's core protocol logic. +* [tig-runtime](./tig-runtime/README.md) - A Rust crate that execute an algorithm (compiled from [`tig-binary`](./tig-binary/README.md)) for a single nonce, generating runtime signature and fuel consumed for verification. * [tig-structs](./tig-structs/README.md) - A Rust crate that contains the definitions of structs used throughout TIG * [tig-token](./tig-token/README.md) - Solidity contract for TIG's ERC20 token that is deployed on Ethereum L2 Base chain * [tig-utils](./tig-utils/README.md) - A Rust crate that contains utility functions used throughout TIG -* [tig-wasm](./tig-wasm/README.md) - A Rust crate for wrapping algorithm submissions for compilation into WASM with an exported `entry_point` -* [tig-worker](./tig-worker/README.md) - A Rust crate for verifying and computing solutions +* [tig-verifier](./tig-verifier/README.md) - A Rust crate that verifies a single solution or Merkle proof. +* [tig-worker](./tig-worker/README.md) - A Rust crate for executing a batch of instances using [`tig-runtime`](./tig-runtime/README.md), aggregating outputs, and calculating Merkle root. ## Useful Scripts diff --git a/docs/guides/innovating.md b/docs/guides/innovating.md index 06768f5..3278002 100644 --- a/docs/guides/innovating.md +++ b/docs/guides/innovating.md @@ -2,9 +2,9 @@ ## Developer Environment -TIG compiles all algorithms into WASM for sandboxing and for verifiable execution. +TIG compiles all algorithms into shared objects and ptx for verifiable execution. -TIG currently requires all algorithms to be written in [Rust](https://www.rust-lang.org/tools/install). +TIG currently requires all algorithms to be written in [Rust](https://www.rust-lang.org/tools/install). Algorithms can optionally have a .cu with Cuda kernels, which are launched from the Rust code. We recommend developing using [Visual Studio Code](https://code.visualstudio.com/) with Rust plugins: * rust-analyzer @@ -12,54 +12,29 @@ We recommend developing using [Visual Studio Code](https://code.visualstudio.com * crates * CodeLLDB -## Setting up Private Fork - -Innovators will want to create a private fork so that they can test that their algorithm can be successfully compiled into WASM by the CI. - -1. Create private repository on GitHub -2. Create empty git repository on your local machine - ``` - mkdir tig-monorepo - cd tig-monorepo - git init - ``` -3. Setup remotes with origin pointed to your private repository - ``` - git remote add origin - git remote add public https://github.com/tig-foundation/tig-monorepo.git - ``` - -4. Pulling `blank_slate` from TIG public repository (branch with no algorithms) - ``` - git fetch public - git checkout -b blank_slate - git pull public blank_slate - ``` - -5. Push to your private repository - ``` - git push origin blank_slate - ``` - ## Checking out Existing Algorithms Every algorithm has its own `` with name `/`. -Only algorithms that are successfully compiled into WASM have their branch pushed to this public repository. +To pull an existing algorithm from tig-monorepo repository, run the following command: +``` +git pull origin +``` -Each algorithm branch will have 6 key files: +Only algorithms that are successfully compiled have their branch pushed to this public repository. + +Each algorithm branch will have 6 key files (11 if there is CUDA code): 1. Rust code with TIG commercial license header @ `tig-algorithms/src//commercial.rs` 2. Rust code with TIG open data license header @ `tig-algorithms/src//open_data.rs` 3. Rust code with TIG benchmarker outbound license header @ `tig-algorithms/src//benchmarker_outbound.rs` 4. Rust code with TIG innovator outbound license header @ `tig-algorithms/src//innovator_outbound.rs` -5. Rust code with TIG inbound license header @ `tig-algorithms/src//inbound.rs` -6. Wasm blob @ `tig-algorithms/wasm/.wasm` - -To pull an existing algorithm from TIG public repository, run the following command: -``` -git fetch public -git pull public -``` +5. Cuda code with TIG inbound license header @ `tig-algorithms/src//inbound.cu` +6. Tarball with shared objects and (optionally) ptx @ `tig-algorithms/lib/.tar.gz` +7. Cuda code with TIG commercial license header @ `tig-algorithms/src//commercial.cu` +8. Cuda code with TIG open data license header @ `tig-algorithms/src//open_data.cu` +9. Cuda code with TIG benchmarker outbound license header @ `tig-algorithms/src//benchmarker_outbound.cu` +10. Cuda code with TIG innovator outbound license header @ `tig-algorithms/src//innovator_outbound.cu` +11. Cuda code with TIG inbound license header @ `tig-algorithms/src//inbound.cu` ## Developing Your Algorithm @@ -67,6 +42,7 @@ git pull public 1. Pick a challenge (``) to develop an algorithm for 2. Make a copy of `tig-algorithms//template.rs` or an existing algorithm (see notes) + * (Optional) for Cuda, additionally make a copy of `tig-algorithms//template.cu` 3. Make sure your file has the following notice in its header if you intend to submit it to TIG: ``` Copyright [year copyright work created] [name of copyright owner] @@ -97,11 +73,14 @@ language governing permissions and limitations under the License. #[cfg(test)] mod tests { - use super::*; - use tig_challenges::{::*, *}; + #[cfg(feature = "cuda")] + use cudarc::{driver::CudaContext, nvrtc::Ptx, runtime::result::device::get_device_prop}; + use super::::solve_challenge; + use tig_challenges::::*; + #[cfg(not(feature = "cuda"))] #[test] - fn test_() { + fn test_algorithm() { let difficulty = Difficulty { // Uncomment the relevant fields. // Modify the values for different difficulties @@ -117,14 +96,14 @@ language governing permissions and limitations under the License. // -- knapsack -- // num_items: 50, // better_than_baseline: 10, - + // -- vector_search -- // num_queries: 10, // better_than_baseline: 350, }; let seed = [0u8; 32]; // change this to generate different instances let challenge = Challenge::generate_instance(seed, &difficulty).unwrap(); - match ::solve_challenge(&challenge) { + match solve_challenge(&challenge) { Ok(Some(solution)) => match challenge.verify_solution(&solution) { Ok(_) => println!("Valid solution"), Err(e) => println!("Invalid solution: {}", e), @@ -133,50 +112,142 @@ language governing permissions and limitations under the License. Err(e) => println!("Algorithm error: {}", e), }; } + + #[cfg(feature = "cuda")] + #[test] + fn test_algorithm() { + let difficulty = Difficulty { + // Uncomment the relevant fields. + // Modify the values for different difficulties + + // -- satisfiability -- + // num_variables: 50, + // clauses_to_variables_percent: 300, + + // -- vehicle_routing -- + // num_nodes: 40, + // better_than_baseline: 250, + + // -- knapsack -- + // num_items: 50, + // better_than_baseline: 10, + + // -- vector_search -- + // num_queries: 10, + // better_than_baseline: 350, + }; + let seed = [0u8; 32]; // change this to generate different instances + + let ptx_path = "/app/tig-algorithms/lib/vector_search/ptx/simple_search.ptx"; // change this to the path of your PTX file + let ptx = Ptx::from_file(ptx_path); + let gpu_device = 0; // change this to the desired GPU device ID + let ctx = CudaContext::new(gpu_device).unwrap(); + ctx.set_blocking_synchronize().unwrap(); + let module = ctx.load_module(ptx).unwrap(); + let stream = ctx.default_stream(); + let prop = get_device_prop(gpu_device as i32).unwrap(); + + let challenge = + Challenge::generate_instance(seed, &difficulty, module.clone(), stream.clone(), &prop) + .unwrap(); + + match solve_challenge(&challenge, module.clone(), stream.clone(), &prop) { + Ok(Some(solution)) => { + match challenge.verify_solution(&solution, module.clone(), stream.clone(), &prop) { + Ok(_) => println!("Valid solution"), + Err(e) => println!("Invalid solution: {}", e), + } + } + Ok(None) => println!("No solution"), + Err(e) => println!("Algorithm error: {}", e), + }; + } } ``` - * See notes for CUDA tests -6. Use the above test to debug your algorithm: +6. Run the appropiate [development docker image](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fdev). Available flavours are: + * amd64 (x86_64 compatible) + * aarch64 + * amd64-cuda12.6.3 (x86_64 compatible) + * aarch64-cuda12.6.3 ``` - cargo test -p tig-algorithms -- --nocapture + # example + docker run -it -v $(pwd):/app --gpus all ghcr.io/tig-foundation/tig-monorepo/dev:0.0.1-amd64-cuda12.6.3 + ``` +6. If you have Cuda code, use `build_ptx.py` to compile it + ``` + build_ptx.py + ``` +7. Run the test + * No cuda: + ``` + cargo test -p tig-algorithms --release -- --nocapture + ``` + * With cuda: + ``` + cargo test -p tig-algorithms --release --features cuda -- --nocapture ``` **IMPORTANT (READ THIS):** -* Not all challenge instances have solutions. Algorithms that can detect such cases and exit early (`return Ok(None)`) will potentially have better performance than algorithms that don't exit early +* Not all challenge instances have solutions. * You can find the current qualifying difficulties by: * Query https://mainnet-api.tig.foundation/get-block for * Query https://mainnet-api.tig.foundation/get-challenges?block_id= for * If you are copying and modifying an algorithm that has been submitted to TIG, make sure to use the `innovator_outbound` version * Do not include tests in your algorithm file. TIG will reject your algorithm submission. -* Only your algorithm's rust code gets submitted. You should not be modifying `Cargo.toml` in `tig-algorithms`. Any extra dependencies you add will not be available when TIG compiles your algorithm +* Only your algorithm's code gets submitted. You should not be modifying `Cargo.toml` in `tig-algorithms`. Any extra dependencies you add will not be available when TIG compiles your algorithm * If you need to use random number generation, ensure that it is seeded so that your algorithm is deterministic. * Suggest to use `let mut rng = SmallRng::from_seed(StdRng::from_seed(challenge.seed).gen())` -## Locally Compiling Your Algorithm into WASM +## Locally Compiling Your Algorithm into Shared Object -See the [README](../../tig-wasm/README.md) for `tig-wasm` +See the [README](../../tig-binary/README.md) for `tig-binary` ## Testing Performance of Algorithms -See the [README](../../tig-worker/README.md) for `tig-worker` +TODO ## Checking CI Successfully Compiles Your Algorithm TIG pushes all algorithms to their own branch which triggers the CI (`.github/workflows/build_algorithm.yml`). -To trigger the CI on your private repo, your branch just needs to have a particular name: -``` -git checkout -b / -git push origin / -``` +To replicate this, Innovators will want to create a private fork: + +1. Create private repository on GitHub +2. Create empty git repository on your local machine + ``` + mkdir tig-monorepo + cd tig-monorepo + git init + ``` +3. Setup remotes with origin pointed to your private repository + ``` + git remote add origin + git remote add public https://github.com/tig-foundation/tig-monorepo.git + ``` + +4. Pulling `blank_slate` from TIG public repository (branch with no algorithms) + ``` + git fetch public + git checkout -b blank_slate + git pull public blank_slate + ``` + +5. Push to your private repository + ``` + git push origin blank_slate + ``` + +6. Trigger the CI on your private repo with your algorithm branch: + ``` + git checkout -b / blank_slate + git push origin / + ``` ## Making Your Submission - 10 TIG will be deducted from your Available Fee Balance to make a submission. You can topup via the [Benchmarker page](https://play.tig.foundation/benchmarker) **IMPORTANT:** * Submissions are final and cannot be modified after they are made * Be sure to adhere to the notes in [developing your algorithm](#developing-your-algorithm) -* We highly recommend [compiling your algorithm into WASM](#locally-compiling-your-algorithm-into-wasm) and [testing its performance](#testing-performance-of-algorithms) with `tig-worker`. * We highly recommend [checking the CI can compile your algorithm](#checking-ci-successfully-compiles-your-algorithm) diff --git a/tig-algorithms/README.md b/tig-algorithms/README.md index ec0e769..a2015b7 100644 --- a/tig-algorithms/README.md +++ b/tig-algorithms/README.md @@ -12,9 +12,9 @@ See [the guide](../docs/guides/innovating.md) ## Downloading an Algorithm -WASM blobs for an algorithm are stored in the `wasm` subfolder and can be downloaded via: +Shared objects and ptx for an algorithm are stored in the `lib/` subfolder and can be downloaded via: -`https://raw.githubusercontent.com/tig-foundation/tig-monorepo//tig-algorithms/wasm/.wasm` +`https://raw.githubusercontent.com/tig-foundation/tig-monorepo//tig-algorithms/lib/.tar.gz` ## Algorithm Submission Flow @@ -30,8 +30,8 @@ WASM blobs for an algorithm are stored in the `wasm` subfolder and can be downlo Each algorithm submission will have 5 versions, each under a specific license: -* `commercial.rs` will be under TIG commercial license -* `open_data.rs` will be under TIG open data license -* `benchmarker_outbound.rs` will be under TIG benchmarker outbound license -* `innovator_outbound.rs` will be under TIG innovator outbound license -* `inbound.rs` will be under TIG inbound license \ No newline at end of file +* `commercial` will be under TIG commercial license +* `open_data` will be under TIG open data license +* `benchmarker_outbound` will be under TIG benchmarker outbound license +* `innovator_outbound` will be under TIG innovator outbound license +* `inbound` will be under TIG inbound license \ No newline at end of file diff --git a/tig-benchmarker/README.md b/tig-benchmarker/README.md index b5be57e..b19c734 100644 --- a/tig-benchmarker/README.md +++ b/tig-benchmarker/README.md @@ -43,14 +43,20 @@ See last section on how to find your player_id & api_key. # Connecting Slaves -1. Compile `tig-worker`: +1. Run the appropiate [runtime docker image](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fruntime) for your slave. Available flavours are: + * amd64 (x86_64 compatible) + * aarch64 + * amd64-cuda12.6.3 (x86_64 compatible) + * aarch64-cuda12.6.3 ``` - cargo build -p tig-worker --release + # example + docker run -it --gpus all ghcr.io/tig-foundation/tig-monorepo/runtime:0.0.1-amd64-cuda12.6.3 ``` + 2. Run `slave.py`: ``` - # assume you are in tig-benchmarker folder - python3 slave.py ../target/release/tig-worker + # runtime docker container should start you in /app + python3 slave.py --help ``` **Notes:** @@ -59,38 +65,9 @@ See last section on how to find your player_id & api_key. * To use a different port, use the option `--port ` * To see all options, use `--help` -## Hard Resetting Your Slave - -1. Stop your slave -2. Remove the output folder (defaults to results): `rm -rf results` -3. Start your slave - # Optimising your Config -1. `difficulty_sampler_config` allows you to configure the difficulty your benchmarker selects for each challenge: - * `selected_difficulties` expects a list of difficulties. If any of those difficulties are within the valid range, they will be picked - * `difficulty_ranges` expects a range defined by 2 numbers: `[lower, higher]`. This is used to sample a random difficulty in the valid range. Examples: - * `[0.0, 1.0]` samples the full range of valid difficulties - * `[0.0, 0.1]` samples the easiest 10% of valid difficulties - - -2. `job_manager_config` allows you to set the `batch_size` for each challenge. - * `batch_size` is the number of nonces that are part of a batch. Must be a power of 2 - * Recommend to pick a `batch_size` for your slave with lowest `num_workers` such that it takes a few seconds to compute (e.g. 5 seconds) - * `batch_size` shouldn't be too small, or else network latency between master and slave will affect performance - * Use `slave_manager_config` to support slaves with different `num_workers` - -3. `precommit_manager_config` allows you to control your benchmarks: - * `max_pending_benchmarks` is the maximum number of pending benchmarks. You dont want benchmarks to take too long, nor do you want your slaves to be idle too much - - * `num_nonces` is the number of nonces to compute per benchmark. See Discord #community-tools channel for stats to help pick - - * `weight` affects how likely the challenge will be picked (weight of 0 will never be picked). Observe your cutoff and imbalance on the [benchmarker page](https://play.tig.foundation/benchmarker) - -4. `slave_manager_config` allows you to control your slaves: - * Slave names are matched to regexes (or else rejected). - * `max_concurrent_batches` is the max number of concurrent batches a slave can work on - * `selected_challenges` is a whitelist of challenges for that slave. If you don't want a slave to benchmark a specific challenge, remove its entry from the list +See [docs.tig.foundation](https://docs.tig.foundation/benchmarking/benchmarker-config) # Finding your API Key @@ -98,17 +75,13 @@ See last section on how to find your player_id & api_key. 1. Navigate to https://play.tig.foundation/ 2. Connect your wallet -3. Run the following command in the console: `JSON.parse(Cookies.get("account"))` - * `address` is your Mainnet `player_id` - * `api_key` is your Mainnet API key +3. Your API key can be copied from the bottom left corner of the dashboard ## Testnet 1. Navigate to https://test.tig.foundation/ 2. Connect your wallet -3. Run the following command in the console: `JSON.parse(Cookies.get("account"))` - * `address` is your Testnet `player_id` - * `api_key` is your Testnet API key +3. Your API key can be copied from the bottom left corner of the dashboard # License diff --git a/tig-benchmarker/common/structs.py b/tig-benchmarker/common/structs.py index 9119ce7..39fa53d 100644 --- a/tig-benchmarker/common/structs.py +++ b/tig-benchmarker/common/structs.py @@ -103,6 +103,7 @@ class OutputData(FromDict): runtime_signature: int fuel_consumed: int solution: dict + cpu_arch: str def calc_solution_signature(self) -> int: return u64s_from_str(jsonify(self.solution))[0] diff --git a/tig-binary/README.md b/tig-binary/README.md new file mode 100644 index 0000000..66c6c16 --- /dev/null +++ b/tig-binary/README.md @@ -0,0 +1,39 @@ +# tig-binary + +A Rust crate that wraps an algorithm from [`tig-algorithm`](../tig-algorithms/README.md) for compilation into a shared object. + +TIG uses a forked version of [LLVM compiler](https://github.com/tig-foundation/llvm) for injecting runtime signature and fuel tracking into the shared object. + +For CUDA, TIG uses `nvcc` to generate ptx using target version `sm_70/compute_70`, before injecting runtime signature and fuel tracking into the ptx file. + +# Getting Started + +1. Run the appropiate [development docker image](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fdev). Available flavours are: + * amd64 (x86_64 compatible) + * aarch64 + * amd64-cuda12.6.3 (x86_64 compatible) + * aarch64-cuda12.6.3 +``` +# example +docker run -it -v $(pwd):/app ghcr.io/tig-foundation/tig-monorepo/dev:0.0.1-aarch64 +# scripts build_so.sh and build_ptx.py are on PATH +``` + +2. Build shared object using `build_so.sh` script: + * Expects `tig_algorithm::::::solve_challenge` to be importable + * Outputs to `tig-algorithms/lib///.so`, where `ARCH` is aarch64 or amd64 + ``` + # add '--cuda' flag if building cuda algorithm + build_so.sh $CHALLENGE $ALGORITHM + ``` + +3. (Optional) Build ptx using `build_ptx.py` script: + * Expects `tig_algorithm/src//.cu` or `tig_algorithm/src///benchmarker_outbound.cu` file to exist + * Outputs to `tig-algorithms/lib//ptx/.ptx` +``` +build_ptx.py $CHALLENGE $ALGORITHM +``` + +# License + +[End User License Agreement](../docs/agreements/end_user_license_agreement.pdf) \ No newline at end of file diff --git a/tig-runtime/README.md b/tig-runtime/README.md new file mode 100644 index 0000000..4d0fc41 --- /dev/null +++ b/tig-runtime/README.md @@ -0,0 +1,97 @@ +# tig-runtime + +A Rust crate that execute an algorithm (compiled from [`tig-binary`](../tig-binary/README.md)) for a single nonce, generating runtime signature and fuel consumed for verification. + +# Getting Started + +Users who don't intend to customise `tig-runtime` are recommended to download pre-compiled version available in [TIG's runtime docker images](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fruntime). + +**Example:** +``` +docker run -it ghcr.io/tig-foundation/tig-monorepo/runtime:0.0.1-aarch64 +# tig-runtime is already on PATH +``` + +## Compiling (using dev docker image) + +The required rust environment for development are available via [TIG's development docker images](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fdev). + + +**Example:** +``` +docker run -it -v $(pwd):/app ghcr.io/tig-foundation/tig-monorepo/dev:0.0.1-aarch64 +# cargo build -p tig-runtime --release +``` + +## Compiling (local setup) + +Users who intend to customise `tig-runtime` need to install a specific version of rust: + +1. Install rust version `nightly-2025-02-10` +``` +ARCH=$(uname -m) +RUST_TARGET=$(if [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then + echo "aarch64-unknown-linux-gnu"; +else + echo "x86_64-unknown-linux-gnu"; +fi) +rustup install nightly-2025-02-10 +rustup default nightly-2025-02-10 +rustup component add rust-src +rustup target add $RUST_TARGET +RUST_LIBDIR=$(rustc --print target-libdir --target=$RUST_TARGET) +ln -s $RUST_LIBDIR /usr/local/lib/rust +echo "export LD_LIBRARY_PATH=\"${LD_LIBRARY_PATH}:/usr/local/lib/rust\"" >> ~/.bashrc +``` + +2. Compile `tig-runtime` +``` +# for cuda version, add --features cuda +cargo build -p tig-runtime --release --target $RUST_TARGET +``` + +# Usage + +``` +Usage: tig-runtime [OPTIONS] + +Arguments: + Settings json string or path to json file + A string used in seed generation + Nonce value + Path to a shared object (*.so) file + +Options: + --ptx [] Path to a CUDA ptx file + --fuel [] Optional maximum fuel parameter [default: 2000000000] + --output [] If set, the output data will be saved to this file path (default json) + --compress [] If output file is set, the output data will be compressed as zlib [default: false] [possible values: true, false] + --gpu [] Which GPU device to use [default: 0] + -h, --help Print help +``` + +The following exit codes indicate specific meanings: +* 0 - solution found +* 85 - no solution found +* 86 - runtime error +* 87 - out of fuel + +**Example:** +``` +SETTINGS='{"challenge_id":"c001","difficulty":[50,300],"algorithm_id":"","player_id":"","block_id":""}' +RANDHASH='rand_hash'$ +NONCE=1337 +FUEL=987654321123456789 +SO_PATH=./tig-algorithms/lib/satisfiability/aarch64/better_sat.so + +tig-runtime $SETTINGS $RANDHASH $NONCE $SO_PATH --fuel $FUEL +``` + +**Example Output:** +``` +{"cpu_arch":"aarch64","fuel_consumed":95496,"nonce":1337,"runtime_signature":4125818588297548058,"solution":{"variables":[0,1,0,1,0,0,0,0,0,1,0,0,0,1,1,1,0,1,1,1,0,0,0,0,1,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,1,1,1,0,0]}} +``` + +# License + +[End User License Agreement](../docs/agreements/end_user_license_agreement.pdf) \ No newline at end of file diff --git a/tig-runtime/src/main.rs b/tig-runtime/src/main.rs index c682c3a..fa3e73a 100644 --- a/tig-runtime/src/main.rs +++ b/tig-runtime/src/main.rs @@ -42,7 +42,7 @@ fn cli() -> Command { .value_parser(clap::value_parser!(u64)), ) .arg( - arg!(--output [OUTPUT_FILE] "If set, the output data will be saved to this file path (default json") + arg!(--output [OUTPUT_FILE] "If set, the output data will be saved to this file path (default json)") .value_parser(clap::value_parser!(PathBuf)), ) .arg( diff --git a/tig-verifier/README.md b/tig-verifier/README.md new file mode 100644 index 0000000..1372c9b --- /dev/null +++ b/tig-verifier/README.md @@ -0,0 +1,89 @@ +# tig-verifier + +A Rust crate that verifies a single solution or Merkle proof. + +# Getting Started + +Users who don't intend to customise `tig-verifier` are recommended to download pre-compiled version available in [TIG's runtime docker images](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fruntime). + +**Example:** +``` +docker run -it ghcr.io/tig-foundation/tig-monorepo/runtime:0.0.1-aarch64 +# tig-verifier is already on PATH +``` + +## Compiling (using dev docker image) + +The required rust environment for development are available via [TIG's development docker images](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fdev). + + +**Example:** +``` +docker run -it -v $(pwd):/app ghcr.io/tig-foundation/tig-monorepo/dev:0.0.1-aarch64 +# cargo build -p tig-verifier --release +``` + +## Compiling (local setup) + +Users who intend to customise `tig-verifier` need to install a specific version of rust: + +1. Install rust version `nightly-2025-02-10` +``` +ARCH=$(uname -m) +RUST_TARGET=$(if [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then + echo "aarch64-unknown-linux-gnu"; +else + echo "x86_64-unknown-linux-gnu"; +fi) +rustup install nightly-2025-02-10 +rustup default nightly-2025-02-10 +rustup component add rust-src +rustup target add $RUST_TARGET +RUST_LIBDIR=$(rustc --print target-libdir --target=$RUST_TARGET) +ln -s $RUST_LIBDIR /usr/local/lib/rust +echo "export LD_LIBRARY_PATH=\"${LD_LIBRARY_PATH}:/usr/local/lib/rust\"" >> ~/.bashrc +``` + +2. Compile `tig-verifier` +``` +# for cuda version, add --features cuda +cargo build -p tig-verifier --release --target $RUST_TARGET +``` + +# Usage + +`tig-verifier` will have multiple sub-commands, but currently only `verify_solution` is implemented: + +``` +Usage: tig-verifier verify_solution [OPTIONS] + +Arguments: + Settings json string or path to json file + A string used in seed generation + Nonce value + Solution json string, path to json file, or '-' for stdin + +Options: + --ptx [] Path to a CUDA ptx file + --gpu [] Which GPU device to use [default: 0] + -h, --help Print help +``` + +**Example:** +``` +SETTINGS='{"challenge_id":"c001","difficulty":[50,300],"algorithm_id":"","player_id":"","block_id":""}' +RANDHASH='rand_hash' +NONCE=1337 +SOLUTION='{"variables":[1,1,0,0,0,0,1,0,0,1,0,0,0,1,0,0,0,1,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0]}' + +tig-verifier verify_solution $SETTINGS $RANDHASH $NONCE $SOLUTION +``` + +**Example Output:** +``` +Solution is valid +``` + +# License + +[End User License Agreement](../docs/agreements/end_user_license_agreement.pdf) \ No newline at end of file diff --git a/tig-worker/README.md b/tig-worker/README.md index 3a26d60..5715c6c 100644 --- a/tig-worker/README.md +++ b/tig-worker/README.md @@ -1,51 +1,88 @@ # tig-worker -A Rust crate for verifying and computing solutions. +A Rust crate for executing a batch of instances using [`tig-runtime`](../tig-runtime/README.md), aggregating outputs, and calculating Merkle root. -Solutions are computed by executing an algorithm in a WASM virtual machine ([TIG's fork of wasmi](https://github.com/tig-foundation/wasmi)). +# Getting Started -# Compiling +`tig-worker` executes a number of `tig-runtime` concurrently. Each `tig-runtime` loads an algorithm shared object (compiled from `tig-binary`), which expects a specific version of rust standard libraries to be available on `LD_LIBRARY_PATH`. +Users who don't intend to customise `tig-worker` are recommended to download pre-compiled version available in [TIG's runtime docker images](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fruntime). + +**Example:** ``` -cargo build -p tig-worker --release -./target/release/tig-worker --help +docker run -it ghcr.io/tig-foundation/tig-monorepo/runtime:0.0.1-aarch64 +# tig-worker is already on PATH +``` + +## Compiling (using dev docker image) + +The required rust environment for development are available via [TIG's development docker images](https://github.com/tig-foundation/tig-monorepo/pkgs/container/tig-monorepo%2Fdev). + + +**Example:** +``` +docker run -it -v $(pwd):/app ghcr.io/tig-foundation/tig-monorepo/dev:0.0.1-aarch64 +# cargo build -p tig-worker --release +``` + +## Compiling (local setup) + +Users who intend to customise `tig-worker` need to install a specific version of rust: + +1. Install rust version `nightly-2025-02-10` +``` +ARCH=$(uname -m) +RUST_TARGET=$(if [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then + echo "aarch64-unknown-linux-gnu"; +else + echo "x86_64-unknown-linux-gnu"; +fi) +rustup install nightly-2025-02-10 +rustup default nightly-2025-02-10 +rustup component add rust-src +rustup target add $RUST_TARGET +RUST_LIBDIR=$(rustc --print target-libdir --target=$RUST_TARGET) +ln -s $RUST_LIBDIR /usr/local/lib/rust +echo "export LD_LIBRARY_PATH=\"${LD_LIBRARY_PATH}:/usr/local/lib/rust\"" >> ~/.bashrc +``` + +2. Compile `tig-worker` +``` +# for cuda version, add --features cuda +cargo build -p tig-worker --release --target $RUST_TARGET ``` # Usage -`tig-worker` has sub-commands `verify_solution`, `compute_solution` and `compute_batch`. These are used in 2 scripts: - -* [Test algorithm performance](../scripts/test_algorithm_performance.sh) -* [Verify benchmark solutions](../scripts/verify_benchmark_solutions.sh) - -## Verify Solution - -Given settings, nonce and a solution, `tig-worker` verifies the solution is a valid solution for the challenge instance. - -* If the solution is valid, `tig-worker` will terminate with exit code 0 - -* If the solution is invalid, `tig-worker` will terminate with exit code 1 - ``` -Usage: tig-worker verify_solution +Usage: tig-worker [OPTIONS] Arguments: - Settings json string or path to json file - A string used in seed generation - Nonce value - Solution json string or path to json file + Path to tig-runtime executable + Settings json string or path to json file + A string used in seed generation + Starting nonce + Number of nonces to compute + Batch size for Merkle tree + Path to a shared object (*.so) file Options: - -h, --help Print help + --ptx [] Path to a CUDA ptx file + --fuel [] Optional maximum fuel parameter for runtime [default: 2000000000] + --workers [] Number of worker threads [default: 1] + --output [] If set, the data for nonce will be saved as '.json' in this folder + -h, --help Print help ``` **Example:** ``` -SETTINGS='{"challenge_id":"c001","difficulty":[50,300],"algorithm_id":"","player_id":"","block_id":""}' -NONCE=7 -SOLUTION='{"variables":[1,0,0,0,0,0,1,0,0,0,1,1,1,1,0,1,1,0,0,1,1,1,1,0,1,0,1,1,0,0,1,1,0,0,0,0,1,0,0,1,1,1,1,1,1,0,1,1,1,0]}' -RAND_HASH=random_string -./target/release/tig-worker verify_solution $SETTINGS $RAND_HASH $NONCE $SOLUTION +SETTINGS='{"challenge_id":"c001","difficulty":[5000,415],"algorithm_id":"","player_id":"","block_id":""}' +RANDHASH='rand_hash'$ +NONCE=1337 +FUEL=987654321123456789 +SO_PATH=./tig-algorithms/lib/satisfiability/aarch64/better_sat.so + +tig-worker \tig-runtime $SETTINGS $RANDHASH $NONCE $SO_PATH --fuel $FUEL ``` ## Compute Solution @@ -73,64 +110,21 @@ Options: **Example:** ``` -SETTINGS='{"challenge_id":"c001","difficulty":[50,300],"algorithm_id":"","player_id":"","block_id":""}' -NONCE=7 -WASM=./tig-algorithms/wasm/satisfiability/schnoing.wasm +SETTINGS='{"challenge_id":"c001","difficulty":[5000,415],"algorithm_id":"","player_id":"","block_id":""}' +START=0 +NUM_NONCES=8 +BATCH_SIZE=8 +SO_PATH=./tig-algorithms/lib/satisfiability/aarch64/better_sat.so RAND_HASH=random_string -./target/release/tig-worker compute_solution $SETTINGS $RAND_HASH $NONCE $WASM + +tig-worker \tig-runtime $SETTINGS $RAND_HASH $START $NUM_NONCES $BATCH_SIZE $SO_PATH --workers 8 ``` -# Compute Batch -Computes a batch of nonces and generates Merkle root and proofs. - +**Example Output:** ``` -Usage: tig-worker compute_batch [OPTIONS] - -Arguments: - Settings json string or path to json file - A string used in seed generation - Starting nonce - Number of nonces to compute - Batch size for Merkle tree - Path to a wasm file - -Options: - --fuel [] Optional maximum fuel parameter for WASM VM [default: 2000000000] - --mem [] Optional maximum memory parameter for WASM VM [default: 1000000000] - --sampled ... Sampled nonces for which to generate proofs - --workers [] Number of worker threads [default: 1] - -h, --help Print help +{"merkle_root":"ab3f7ea08a2b991217bd9b08299b063bc77a0239af3a826d3b0ea91ca3384f98","solution_nonces":[6,3,4,5,7,2,1,0]} ``` -**Example:** -``` -SETTINGS='{"challenge_id":"c001","difficulty":[50,300],"algorithm_id":"","player_id":"","block_id":""}' -START_NONCE=0 -NUM_NONCES=1000 -BATCH_SIZE=1024 -WASM=./tig-algorithms/wasm/satisfiability/schnoing.wasm -RAND_HASH=random_string -./target/release/tig-worker compute_batch $SETTINGS $RAND_HASH $START_NONCE $NUM_NONCES $BATCH_SIZE $WASM -``` - -# Notes - -* `challenge_id` must be set: - * `c001` is satisfiability - * `c002` is vehicle_routing - * `c003` is knapsack - * `c004` is vector_search -* Recommended low difficulties for testing are: - * satisfiability [50,300] - * vehicle_routing [40, 250] - * knapsack [50, 10] - * vector_search [10, 350] -* You can query the latest difficulties by using `scripts/list_challenges.sh` -* You can test the performance of an algorithm using `scripts/test_algorithm.sh` -* You can list all active benchmark ids with `scripts/list_benchmark_ids.sh` -* You can get benchmark data with `scripts/list_benchmark_ids.sh` -* You can verify a benchmark's solutions, runtime_signature and fuel_consumed with `scripts/verify_benchmark.sh` - # License [End User License Agreement](../docs/agreements/end_user_license_agreement.pdf) \ No newline at end of file