BIGNUM *BN_generate_prime(BIGNUM *ret, int num, int safe, BIGNUM *add, BIGNUM *rem, void (*callback)(int, int, void *), void *cb_arg);
int BN_is_prime(const BIGNUM *a, int checks, void (*callback)(int, int, void *), BN_CTX *ctx, void *cb_arg);
int BN_is_prime_fasttest(const BIGNUM *a, int checks, void (*callback)(int, int, void *), BN_CTX *ctx, void *cb_arg, int do_trial_division);
If callback is not NULL, it is called as follows:
The prime may have to fulfill additional requirements for use in Diffie-Hellman key exchange:
If add is not NULL, the prime will fulfill the condition p % add == rem (p % add == 1 if rem == NULL) in order to suit a given generator.
If safe is true, it will be a safe prime (i.e. a prime p so that (p-1)/2 is also prime).
The PRNG must be seeded prior to calling BN_generate_prime(). The prime number generation has a negligible error probability.
BN_is_prime() and BN_is_prime_fasttest() test if the number a is prime. The following tests are performed until one of them shows that a is composite; if a passes all these tests, it is considered prime.
BN_is_prime_fasttest(), when called with do_trial_division == 1, first attempts trial division by a number of small primes; if no divisors are found by this test and callback is not NULL, callback(1, -1, cb_arg) is called. If do_trial_division == 0, this test is skipped.
Both BN_is_prime() and BN_is_prime_fasttest() perform a Miller-Rabin probabilistic primality test with checks iterations. If checks == BN_prime_checks, a number of iterations is used that yields a false positive rate of at most 2^-80 for random input.
If callback is not NULL, callback(1, j, cb_arg) is called after the j-th iteration (j = 0, 1, ...). ctx is a pre-allocated BN_CTX (to save the overhead of allocating and freeing the structure in a loop), or NULL.
BN_is_prime() returns 0 if the number is composite, 1 if it is prime with an error probability of less than 0.25^checks, and -1 on error.
The error codes can be obtained by ERR_get_error(3).