Making Transparents

To see how some specialized transparent columns are built, let's look inside Binius transparents. While specialized transparent columns are sufficient in many scenarios, sometimes we want to make a transparent column out of a plain list of values. For example, if we want to implement a lookup table, we'll need a transparent column to hold the values in the lookup table. Here we build a gadget for the simple case we wish to build a transparent column from a slice of arbitrary values.

First let's decide on a signature for the gadget.

fn make_transparent<U, F, FS>(
    builder: &mut ConstraintSystemBuilder<U, F>,
    name: impl ToString,
    values: &[FS],
) -> Result<OracleId, anyhow::Error>
where
    U: PackScalar<F> + PackScalar<FS>,
    F: TowerField + ExtensionField<FS>,
    FS: TowerField,

Apart from the builder, we'll need the values in some subfield FS of the main field F. The function should return the oracle id for the newly created transparent.

Within this function we declare the transparent oracle as

let oracle =  builder.add_transparent(name, mle)?;

where mle is something that holds the FS values to instantiate this oracle. While it is not necessary to understand the mle (short for "multilinear extension"), we must construct it. To do so we write

use binius_core::transparent::multilinear_extension::MultilinearExtensionTransparent;
let mle = MultilinearExtensionTransparent::<_, PackedType<U, F>, _>::from_values(
    packed_values,
)?;

But notice we're passing in packed_values, whereas we're only given value. We must take the FS values vector given and create a vector of packed FS values.

Before we create the packed values, we must calculate how large our packed value vector will be due to packing. The packing width is PackedType::<U, FS>::WIDTH, so if values.len() was divisible by this packing width we'd simply divide. To account for the case of non-divisibility we instead calculate

let packed_length = values.len().div_ceil(PackedType::<U, FS>::WIDTH);

To create the packed vector we can first initialize a vector of packed_length with default packed values.

let mut packed_values = vec![PackedType::<U, FS>::default(); packed_length];

With a simple loop we can use copy the values into the packed values.

for (i, value) in values.iter().enumerate() {
    binius_field::packed::set_packed_slice(&mut packed_values, i, *value);
}

We've been working backwards. To see these steps in forwards order, we'll write the whole function at the bottom of the page. Before we can be done, we, the prover, must create a witness column for the oracle and populate it with values.

if let Some(witness) = builder.witness() {
    let mut entry_builder = witness.new_column::<FS>(oracle);
    let packed_slice: &mut [PackedType<U, FS>] = entry_builder.packed();
    packed_slice.copy_from_slice(&packed_values);
}

First, we create the new column, receiving an EntryBuilder. Next, we called packed on the entry builder to receive a mutable reference to a slice of packed values. Finally, we copy over our packed values to this packed slice.

Finally we can return our newly created transparent oracle as Ok(oracle).

We write the full gadget here for completion. Notice we must actually clone packed_values when creating the mle, because we will need that vector again when copying to the new column.

fn make_transparent<U, F, FS>(
    builder: &mut ConstraintSystemBuilder<U, F>,
    name: impl ToString,
    values: &[FS],
) -> Result<OracleId, anyhow::Error>
where
    U: PackScalar<F> + PackScalar<FS>,
    F: TowerField + ExtensionField<FS>,
    FS: TowerField,
{
    let packed_length = values.len().div_ceil(PackedType::<U, FS>::WIDTH);
    let mut packed_values = vec![PackedType::<U, FS>::default(); packed_length];
    for (i, value) in values.iter().enumerate() {
        binius_field::packed::set_packed_slice(&mut packed_values, i, *value);
    }
 
    use binius_core::transparent::multilinear_extension::MultilinearExtensionTransparent;
    let mle = MultilinearExtensionTransparent::<_, PackedType<U, F>, _>::from_values(
        packed_values.clone(),
    )?;
 
    let oracle = builder.add_transparent(name, mle)?;
 
    if let Some(witness) = builder.witness() {
        let mut entry_builder = witness.new_column::<FS>(oracle);
        let packed_slice: &mut [PackedType<U, FS>] = entry_builder.packed();
        packed_slice.copy_from_slice(&packed_values);
    }
 
    Ok(oracle)
}