Blockchain Integrations Framework

Chainlink Labs provides self-service tools that node operators and blockchain projects can use to deploy Data Feed integrations on EVM networks. These tools are called the Blockchain Integrations Framework (BIF). This framework includes a command line interface (CLI) and is available as a binary file. The binary is not publicly available. During the Alpha release, you must receive this binary directly from Chainlink Labs.

The framework simplifies the following tasks:

Deploy Chainlink nodes into Kubernetes clusters and distribute jobs to them
Run tests that generate reports detailing compatibility between Chainlink Labs’ EVM integration and a given blockchain
Deploy cluster infrastructure required to test OCR for Data Feeds
Perform longer running soak tests for OCR for Data Feeds
Perform smart contract management tasks with the same capabilities as Chainlink Labs’ tools

Requirements

Most basic tests can be completed on a system with minimal hardware resources. To complete soak testing and use the CLI to create clusters with multiple Chainlink nodes, you must run a Kubernetes cluster on a system with Docker and the following system resources:

A Linux system
6 CPU cores
10 GiB of system memory
At least 50 GiB of disk storage

Additionally, Docker’s resource limits must also allow these resources. If you are using Docker Desktop, set your Docker resource limits to make sure they meet the system requirements for using the Blockchain Integrations Framework. Docker Engine has resource limits that might be different from Docker Desktop.

Set up your environment

Install the framework and create an alias:

Obtain the binary file from Chainlink Labs and put it in your local directory.
Make the binary available in /usr/bin so you can run it without the full path: sudo mv ./bif /usr/local/bin/
For certain operating systems and architectures, you may need to install wasmvm, a shared library that BIF imports. For example, on MacOS, you must install libwasmvm.dylib using wget https://raw.githubusercontent.com/CosmWasm/wasmvm/main/internal/api/libwasmvm.dylib and move it to /usr/local/lib using cp libwasmvm.dylib /usr/local/lib.

You can now run the framework CLI. Run bif -h to see the available commands. See the Running basic tests section for examples.

If you plan to run soak tests or other tests that require running Chainlink nodes, set up Helm, Kubernetes and other required tools:

Install the Docker Engine or Docker Desktop.
Set your Docker resource limits to make sure they meet the system requirements for using the Blockchain Integrations Framework. Docker Engine has resource limits that might be different from Docker Desktop.
Install Helm.
Add the following repositories:
- chainlink-qa: helm repo add chainlink-qa https://raw.githubusercontent.com/smartcontractkit/qa-charts/gh-pages/
- bitnami: helm repo add bitnami https://charts.bitnami.com/bitnami
Update the helm repository: helm repo update
Install the kubectl tool.

The chainlink-env repository includes several tools for creating clusters of Chainlink nodes for testing. Use this to start a cluster:

Clone the chainlink-env repository and change directories: git clone https://github.com/smartcontractkit/chainlink-env.git && cd chainlink-env
Install dependencies: make install_deps
Install k3d from k3d.io.
Optionally, install Lens from k8slens.dev or use k9s as a low resource consumption alternative from k9scli.io. These tools offer user-friendly cli to interact with kubernetes clusters. They are not necessary for this tutorial.
Create a cluster: make create_cluster
Install monitoring: make install_monitoring

The make install_monitoring command starts Grafana, which takes control of your terminal. Open Grafana in a browser at localhost:3000 to confirm that it is running properly. Log in with admin as the username and [email protected] as the default password. If you are running testing a remote system, you can use ssh -i $KEY [email protected]$REMOTE-IP -L 3000:localhost:3000 -N to create an SSH tunnel to the system where Grafana is running and map port 3000 to your local workstation. Change the default Grafana password.

You can now run integration testing.

When you are done with testing, you can clean up the environment using the following commands:

Stop the cluster: make stop_cluster
Delete the cluster: make delete_cluster

Running basic tests

You can use the framework to run several basic tests on a network. These commands deploy contracts as part of the tests, so you must provide the following parameters:

The private key to a wallet that is funded and capable of deploying contracts on the network that you want to test
The chain ID for your network, which usually can be found on the LINK Token Contracts or chainlist.org
The RPC URL for either a network node that you run or a third-party service like Alchemy, Infura, and Ankr. Note: Use a websocket URL. Subscriptions require websockets.

Test compatibility of the network client API

bif compatibility test-rpc \
--privateKey <PRIVATE_KEY> \
--chainID <CHAIN_ID> \
--rpc <RPC_URL> \
--storageContractAddress <OPTIONAL_CONTRACT_ADDRESS>

The --storageContractAddress flag is available so you can re-use deployed contracts from previous tests and save wallet funds.

After running the test, you should get a similar output:

Running Compatibility Tests...
Deploying Storage test contract...
Successfully deployed contract to address 0x7d626A527a639Cb0b7e0A29CC8539E210063f77A

API Test Report:

eth_call                                      - chainlink-compatible ✅
eth_estimateGas                               - chainlink-compatible ✅
eth_estimateGas (with block number)           - chainlink-compatible ✅
eth_getCode                                   - chainlink-compatible ✅
eth_getTransactionReceipt                     - chainlink-compatible ✅
eth_getBlockByHash                            - chainlink-compatible ✅
eth_getBlockByNumber                          - chainlink-compatible ✅
eth_getTransactionCount                       - chainlink-compatible ✅
eth_getBalance                                - chainlink-compatible ✅
eth_maxPriorityFeePerGas                      - chainlink-compatible ✅
eth_gasPrice                                  - chainlink-compatible ✅
eth_sendRawTransaction                        - chainlink-compatible ✅
eth_subscribe                                 - chainlink-compatible ✅
eth_getLogs                                   - chainlink-compatible ✅
InsufficientEth                               - chainlink-compatible ✅
NonceTooLow                                   - chainlink-compatible ✅
TransactionAlreadyInMempool                   - chainlink-compatible ✅

Tests Ran Successfully.

Test the smart contract op codes of a network

bif compatibility test-opcodes \
--privateKey <PRIVATE_KEY> \
--chainID <CHAIN_ID> \
--rpc <RPC_URL> \
--opcodesContractAddress <OPTIONAL_CONTRACT_ADDRESS>

The --opcodesContractAddress flag is available so you can re-use deployed contracts from previous tests and save wallet funds.

After running the test, you should get a similar output:

Deploying Opcodes contract...
Successfully deployed contract to address 0xABc3B3cEC3978d831e5585fc6cBCc11B5fEDA2fc
Running Opcode Tests on contract 0xABc3B3cEC3978d831e5585fc6cBCc11B5fEDA2fc...

OpCodes Test Report:

GeneralCodes                                  - chainlink-compatible ✅
Invalid                                       - chainlink-compatible ✅
Stop                                          - chainlink-compatible ✅
Revert                                        - chainlink-compatible ✅

Tests Ran Successfully.

Running integration tests

Integration testing uses the bif integration test command and a TOML file. The TOML file simplifies managing the many variables involved.

The TOML file for the Blockchain Integrations Framework has two settings sections for OCR soak tests on EVM networks:

Nodes settings: Variables that configure chainlink nodes for given networks.
Network settings: Variables that configure the testing client : Which chain you use for testing and the private key for the wallet you want to use for deploying contracts.
Test settings: Test duration, node funding, and round configuration settings to control the behavior of the test.

Example TOML file

# node settings
[nodes]
    [nodes.goerli]

        [nodes.goerli.base]
        eth_chain_id = "5"
        eth_http_url = "https://eth-goerli.g.alchemy.com/v2/<ALCHEMY_KEY>"
        eth_url = "wss://eth-goerli.g.alchemy.com/v2/<ALCHEMY_KEY>"
# network settings
[networks]

    [networks.evm]

        [networks.evm.base]
        name = "EVM"
        evm_keys = ["WALLET_PRIVATE_KEY"]
        ws_urls = []
        http_urls = []
        chain_id = 1337
        simulated = false
        chainlink_transaction_limit = 5000
        transaction_timeout = "2m"
        minimum_confirmations = 1
        gas_estimation_buffer = 1000

        [networks.evm.overrides.goerli]
        name = "Goerli"
        ws_urls = ["wss://eth-goerli.g.alchemy.com/v2/<ALCHEMY_KEY>"]
        chain_id = 5

        [networks.evm.overrides.optimism_goerli]
        name = "Optimism Goerli"
        ws_urls = ["wss://opt-goerli.g.alchemy.com/v2/<ALCHEMY_KEY>"]
        chain_id = 420
        client_implementation = "Optimism"

        [networks.evm.overrides.palm]
        name = "Palm"
        ws_urls = ["wss://palm"]
        chain_id = 11297108099

# test settings
[tests]

    [tests.v1-9-0]

        [tests.v1-9-0.base]
        keep_environments="Never" # Always | OnFail | Never
        # Image repo to pull the Chainlink image from
        chainlink_image="public.ecr.aws/chainlink/chainlink"
        # Version of the Chainlink image to pull
        chainlink_version="1.9.0"
        # Name of the person running the tests (no spaces)
        chainlink_env_user="My-Name"
        test_log_level="info" # info | debug | trace
        node_count=6
        test_duration=15 # minutes
        contract_count=2
        node_funding=0.02 # ETH
        round_timeout=15 # minutes
        expected_round_time=2 # minutes
        time_between_rounds=1 # minutes

        [tests.v1-9-0.overrides.shorter-test]
        test_duration=2 # minutes

If you completed the full steps to set up your environment, you can run a soak test using the following steps:

Create a file called example.toml and paste the example TOML file above.
Edit the file to add your evm_keys under the networks.evm.base config.
Modify the ws_urls under networks.evm.overrides.goerli with your specified RPC endpoint URL. Optionally, you can create additional overrides for different networks.
Start running a soak test bif integration test ./example.toml.
The interactive menu opens. Select your desired:
1. Test suite: Only OCR is available for the moment.
2. Test: Only EVMSoak is available for the moment.
3. Network settings (see your TOML config): evm .
4. Network override (see your TOML config): goerli.
5. Test settings (see your TOML config): v1-9-0.
6. Test override (see your TOML config): shorter-test.
7. The cluster to run on.
While your test is running, view your local Grafana instance at localhost:3000 to see test metrics.
Check node logs: bif integration logs [--level LOG_LEVEL]

If the test is successful, you will see output similar to the following example:

3:56AM INF Soak test concluded
------------------------------
• [SLOW TEST] [434.148 seconds]
OCR Soak Test @ocr-soak With soak test contracts deployed runs the soak test until error or timeout
------------------------------

Ran 1 of 1 Specs in 434.148 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped

As you test additional networks, modify the TOML file and configure it with the details for your network.

Troubleshooting integration tests

There are some common issues with soak testing that you can troubleshoot.

Pods remaining from failed tests:

If a soak test fails, the pods used for testing often remain running. Use kubectl to identify and delete these pods:

List all pods by their namespaces: kubectl get pods --all-namespaces
Delete the deployments that you no longer need by namespace: kubectl delete --all deployments --namespace=ocr-soak-goerli-50b0c
Modify your test configuration and run bif integration soak ./example.toml again.

Insufficient resources:

Your system must meet the defined Requirements. Running on systems without sufficient resources will cause soak tests to fail because some containers for Chainlink nodes will not be able to start.

If you are running on a virtual machine, add additional vCPUs, system memory, or storage.
Run your soak tests with fewer nodes. The default node count is 6 per cluster, which is recommended for a thorough soak test. You can adjust this value if you are unable to obtain the necessary system resources.

Insufficient wallet funding

The soak tests send 0.1 ETH to each test node by default, so you must make sure that there are significant funds in the wallet that you use to fund testing. This is the wallet specified by your evm_keys parameter in the TOML file. Alternatively, you can adjust the node_funding parameter in the TOML test configuration, but make sure that each node has enough funds to complete the soak test.