I am running SCEP client to enroll certificates on NDES server. If OpenSSL is not in FIPS mode - everything works fine.
In FIPS mode i get the following error:
pkcs7_unwrap():pkcs7.c:708] error decrypting inner PKCS#7
139968442623728:error:060A60A3:digital envelope routines:FIPS_CIPHERINIT:disabled for fips:fips_enc.c:142:
139968442623728:error:21072077:PKCS7 routines:PKCS7_decrypt:decrypt error:pk7_smime.c:557:
That's because NDES server uses DES algorithm for encryption returned PKCS#7 packet. I used the following debug code:
/* Copy enveloped data from PKCS#7 */
bytes = BIO_read(pkcs7bio, buffer, sizeof(buffer));
BIO_write(outbio, buffer, bytes);
p7enc = d2i_PKCS7_bio(outbio, NULL);
/* Get encryption PKCS#7 algorithm */
enc_alg=p7enc->d.enveloped->enc_data->algorithm;
evp_cipher=EVP_get_cipherbyobj(enc_alg->algorithm);
printf("evp_cipher->nid = %d\n", evp_cipher->nid);
The last string always prints:
evp_cipher->nid = 31
defined in openssl-1.0.1c/include/openssl/objects.h
#define SN_des_cbc"DES-CBC"
#define LN_des_cbc"des-cbc"
#define NID_des_cbc31
I use 3DES algorithm for PKCS7 requests encryption in my code (pscep.enc_alg = (EVP_CIPHER *)EVP_des_ede3_cbc()) and NDES server accepts these requests, but it always returns answer encrypted with DES.
Can I configure Wndows NDES server to use Triple DES (3DES) algorithm for PKCS#7 answer encryption?