1.5.0: * Introduced smtp acl list, and obsoleted tcp_wrappers * Parse Sender: header and give it preference over From: header * pilerimport puts legacy emails to a top level dir based on their Date: header * pilerexport may digitally sign the exported zip file openssl genpkey -out /etc/piler/signkey.priv -algorithm rsa -pkeyopt rsa_keygen_bits:4096 openssl rsa -in /etc/piler/signkey.priv -pubout -out /etc/piler/signkey.pub chown piler:piler /etc/piler/signkey.* sign: pilerexport -W customer1 -f some@user -z emails.zip -p /etc/piler/signkey.priv verify: openssl dgst -sha256 -verify /etc/piler/signkey.pub -signature emails.zip.sig emails.zip See the following for more: https://www.zimuel.it/blog/sign-and-verify-a-file-using-openssl https://opensource.com/article/19/6/cryptography-basics-openssl-part-2 * pilerexport may export EML files to a zip file. Use -z , eg. pilerexport -W customer1 -f some@user -z first100.zip Note that piler user must be able to create the zip file at the given path. * Added RFC3161 timestamp support Run the following sql query on all piler related databases: alter table customer_settings add column `timestamp` varchar(6) default null; * Preview body of email by hovering mouse over subject lines of the search results * Added support for custom date format and timezone for users Run the following sql query on all piler related databases: alter table user_settings add column date_format char(12) default 'Y.m.d'; alter table user_settings add column timezone varchar(32) default null; * Last search is saved in a cookie, and restored when the user logs in again. * Exclude any email address longer than 41 characters. SphinxQL breaks the query in case of any email 42+ characters long. * Added support for zipped store directories. You may consolidate the zillions of .m files in /var/piler/store// to a series of zip files, eg. /var/piler/store/somecustomer/00/5f3/00 -> /var/piler/store/somecustomer/00/5f3/5f3_00.zip /var/piler/store/somecustomer/00/5f3/01 -> /var/piler/store/somecustomer/00/5f3/5f3_01.zip ... To enable the feature set consolidated_store=1 in /etc/piler/piler.conf Then run /usr/libexec/piler/compact-store-dir.sh 5f3 "somecustomer" Note that you may compact any top level dir except the last one which is currently used. * Improved the encryption resilience by prepending the data to be encrypted with a block of garbage * Switched from Blowfish encryption to AES-256. It's backward compatible. Fix the attachment table in all piler related databases: alter table attachment add column `e` char(1) default 'b'; * IMAP import over the gui To enable the feature for admin users set the following in /etc/piler/config-site.php: $config['ENABLE_IMPORT'] = 1; Also fix the import table in all piler related databases: alter table import add column `customer` varchar(255) default null; If you have a multinodes layout, then run the following command on the worker node which you want to run import jobs on (otherwise it won't process the import jobs): touch /etc/piler/IMPORT_HOST * Discard short header only emails not having a Message-ID: line to prevent them filling up the error directory. 1.4.9: * New customer form supports specifying the admin@local password for the customer at the 'Admin user password' input field. * Support for sphinx-3.3.1 If you upgrade piler, then be sure to set the following in /etc/piler/config-site.php to keep using the legacy full text search columns in sphinx: $config['SPHINX_STRICT_SCHEMA'] = 0; Note that enabling SPHINX_STRICT_SCHEMA (by setting it to 1) breaks the sphinx index data, and they must be recreated with reindex which takes some time. * Added support for multiple master nodes Add the following line to /etc/piler/config-site.php on the current master/gui node if you have a multinode layout: $config['PRIMARY_MASTER'] = 1; To setup an additional, secondary master node set the following to config-site.php: $config['NODE_TYPE'] = MASTER; $config['PRIMARY_MASTER'] = 0; $config['DB_HOSTNAME'] = ''; $config['MASTER_NODE'] = ''; $memcached_server = ['', 11211]; In the above example the secondary master will use the mysql and memcached servers running on the primary master node. * Fixed space projection calculation * Added zipkin compatible tracing support for the GUI * Don't create a default auditor@local account when creating a new tenant/customer * Fixed unique constraint on accounting table: - UNIQUE (`date`, `email`) + UNIQUE (`date`, `email`, `server_id`) * Fixed open_database() call to add mysqlhost parameter * Fixed a typo in r.php * Online users are written to and read from memcached exclusively * Disable prometheus metrics by default. To enable set the following in /etc/piler/config-site.php: $config['ENABLE_PROMETHEUS_METRICS'] = 1; Also it's recommended to set some sort of access control to nginx config, eg. location = /metrics.php { # Add the IP-address of your Prometheus server allow 10.1.1.2; deny all; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_split_path_info ^(.+?\.php)(/.*)$; if (!-f $document_root$fastcgi_script_name) { return 404; } fastcgi_pass unix:PHP_FPM_SOCKET; fastcgi_index index.php; include fastcgi_params; } 1.4.8: * Fixed a bug causing bogus customer info the smtp session * Added piler-s3 systemctl service * Updated cdn links in layout templates, and the matching nginx config Be sure to update the Content-Security-Policy definition: add_header Content-Security-Policy "default-src 'self' 'unsafe-inline' ; object-src 'none'; base-uri 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' code.jquery.com stackpath.bootstrapcdn.com; style-src 'self' 'unsafe-inline' stackpath.bootstrapcdn.com; font-src 'self' stackpath.bootstrapcdn.com; "; * Introduced an improved mobile search template Set $config['ENABLE_MOBILE_PREVIEW'] = 1; in config-site.php to enable it. * Fixed a bug in the health-check.sh script in a multinodes layout. It mistakenly reported a worker -> master probe result * Introduced the Data Officer role You need to create the following table in all piler related databases: create table if not exists `deleted` ( `id` bigint unsigned not null unique, `requestor` varchar(128) not null, `reason1` varchar(255) not null, `date1` int unsigned default 0, `approver` varchar(255) default null, `reason2` varchar(255) default null, `date2` int unsigned default 0, `deleted` tinyint(1) default -1, key (`id`), key (deleted) ) Engine=InnoDB; Add the following rewrite to the webserver config /etc/piler/piler-nginx.conf, then reload the webserver: rewrite /rejectremove.php /index.php?route=message/rejectremove; * Using bootstrap 3.4.1 minified js from CDN 1.4.7: * Fixed customer id check * Fixed creating local users in multitenancy mode * Added backup codes for Google Authenticator You need to create the following table in all piler related databases: create table if not exists `ga_backup_code` ( `username` varchar(64) not null, `code` char(6) not null, `used` tinyint default 0 ) Engine=InnoDB; create index ga_backup_code_idx1 on ga_backup_code(username); * Added systemd service files to /usr/libexec/piler * S3 stuff uses /var/piler/s3 directory mkdir /var/piler/s3 chown piler:piler /var/piler/s3 chmod 700 /var/piler/s3 * S3 upload script (s3up.py) upgraded to use python3. You need the minio python module to use it, eg. pip3 install minio * imapfetch.py and pilerpurge.py uses /usr/bin/python3 as well * Introduced the search history feature Set ENABLE_MEMCACHED in either /etc/piler/config-site.php or in the per customer settings, and be sure to install the php-memcached package. It puts the search terms to memcached to offers them in reverse order, latest on top. * Fixed Google authenticator for ldap logins * Introduced Auth0 support for Azure Set ENABLE_AUTH0 and the appropriate AUTH0 variables in either /etc/piler/config-site.php or in the per customer settings * Default Sphinx version is set to 311 in /var/piler/www/config.php If you have 2.2.x version of sphinx, be sure to add the following line to /etc/piler/config-site.php: $config['SPHINX_VERSION'] = 220; * Added tika and disk info to node health status output * Introduced the default SPHINX_WORKER_LISTEN_ADDRESS variable to set the sphinx listen address on the worker nodes $config['SPHINX_WORKER_LISTEN_ADDRESS'] = '0.0.0.0'; 1.4.6: * The gui provides output for prometheus at /metrics Upgrade: - Add the following to piler's crontab on the worker nodes: 10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors 1.4.5: * Introduced new config variable USE_SMTP_GATEWAY 1.4.4: * Fixed ldap authentication bug (1f828af) * ldap table has new field (ldap_port) defaults to 389 (b094020) Run the following sql statement on all piler related databases: alter table ldap add column `ldap_port` int default 389;