1.8.4:

* Docker image doesn't ship manticore any longer. It shall
  run as a separate container.

  The preferred method is to use real time index tables (RT=1)
  with dynamic index settings (DYNAMIC_INDEX=1)

* Introduced $config1['MANTICORE_WORKER'] variable for
  multinodes environment. Add this to /etc/piler/config-site.php
  The content is similar to $config1['WORKER'] except this
  one lists the manticore nodes. If manticore is installed
  on the worker nodes, then $config1['MANTICORE_WORKER']
  value should be the same as  $config1['WORKER'].

* Introduced $config['MAX_DOWNLOAD_MESSAGES'] variable to
  limit how many emails a user can download by selecting
  them in the left pane. Set to 0 in /etc/piler/config-site.php
  to disable this feature and let the user download as many
  as he likes. The default value is 500.

  Note that the first N messages are selected based on the order
  the user selects them.

* Introduced $config['MAX_EXPORT_MESSAGES'] variable to
  limit how many emails a user can export from a search query.
  Set to 0 in /etc/piler/config-site.php to disable this feature
  and let the user export all emails that matches his query.
  The default value is 20000,

  Note that the first N messages are selected based on the sort
  order the user has set (by default in descending order by sent
  date).

* You may apply IMAP flags to restored emails, eg.
  $config['IMAP_FLAGS'] = ['\Seen'];

* Introduced tika_timeout as a new piler.conf variable
  It sets the time how long the parser waits in seconds
  for tika to return the extracted text. The default
  value is 30 seconds.
  Add the following to /etc/piler/piler.conf:
  tika_timeout=30

* PDF export shall include the message headers (from, to, cc, subject, date)

* Fixed PID file path to /var/piler/run/piler.pid
  Edit /etc/piler/piler.conf and fix the pidfile settings to

  pidfile=/var/piler/run/piler.pid

* Use libcurl for pilerimport imap download




1.8.3:

* Fixed import from GUI in a multinodes environment

* Breaking change! Audit is moved to manticore real time index
  The feature also requires PHP 8.x

* Added mailcow support

  To use it
  - setup imap authentication
  - create a read-only mailcow api key
  - set the following variables in /etc/piler/config-site.php:
     $config['MAILCOW_API_KEY'] = 'xxxx-yyyy-...';
     $config['MAILCOW_HOST'] = 'mailcow.example.com';

* Introduced download-emails-o365.py



1.8.2:

* Introduced 2 new variables in /etc/piler/piler.conf affecting piler-smtp

  ; max message size in bytes
  ; piler-smtp will reject any message that's bigger than this number
  max_message_size=50000000

  ; max memory in bytes piler-smtp uses for buffering messages
  ; when this limit is exceeded, no new emails will be accepted
  ; until the used memory for all in progress emails decreases
  ; below this level
  max_smtp_memory=500000000

  Be sure to adjust these values to your environment!

* Fixed reindex issue when rtindex=1

* If the textual part of the message to display is larger then 100 kB,
  then truncate the displayed content at 100kB.
  To override the threshold value use the following config entry in
  /etc/piler/config-site.php:

  $config['MAX_CONTENT_LENGTH_TO_DISPLAY'] = 100000;

* SAML 2.0 SSO support

  See https://mailpiler.com/saml-2.0-authentication-with-keycloak/ for the details

* Ping identity SSO support

  See https://mailpiler.com/single-sign-on-with-pingidentity/

* Export on the GUI supports >1000 messages

* Auditor may also use the GUI export feature



1.8.1:

* Introduced the redact feature

  See https://mailpiler.com/redact-feature/ for more.

* Fixed tracing issue for api.php

* Added readonly mode for Manticore db connections

  If running sphinx, add the following to /etc/piler/config-site.php:

  $config['SPHINX_HOSTNAME_READONLY'] = '127.0.0.1:9307';

  When upgrading add the following line to the searchd { } block at the
  end of /etc/piler/manticore.conf:

  listen                  = <?php print SPHINX_HOSTNAME_READONLY; ?>:mysql_readonly

* Write manticore pid file to /var/piler/manticore/manticore.pid
  Updated systemd pilersearch.service as well

* Introduced a permission system for actions on messages (view,
  headers, journal, download/restore, export, pdf)

* Fixed a bug in create-customer.sh preventing the restart of pilersearch service

* Introduced categories feature

* Introduced permission profiles

  See https://mailpiler.com/permission-profiles/ for more

* Added support to show a warning to the preview window when displaying an external email
  Note that it requires you to add all your company domains on the gui.

  Also be sure to customize the EXTERNAL_EMAIL_WARNING_TEXT variable, see the default in
  /var/piler/www/config.php. Set the following in /etc/piler/config-site.php to disable this warning:

  $config['EXTERNAL_EMAIL_WARNING_TEXT'] = '';

* Export from the GUI honors the date and attachment settings as well

* Telemetry support. Report an aggregated usage data, in the following format:

  <customer id>/<server id>/<active users>/<received emails>/<size of emails>/<stored size of emails>

  eg. EXAMPLECOM/00/45/122630/435129683/27133196

* Database schema change. Be sure to run the upgrade sql script on
  all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
    mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
  done

* Added encryption support to import table passwords

  Run the following command to install pycryptodome:

  pip install pycryptodome==3.20.0

  Generate an encryption key: openssl rand -base64 24

  Add it to /etc/piler/piler.conf

  crypt_key=..........

  Also add it to /etc/piler/config-site.php:

  $config['ENCRYPTION_KEY'] = '.......';




1.8.0:

* Reworked the GUI using bootstrap 5.3 and vue.js 3

* Be sure to update the rewrite rules and Content-Security-Policy settings
  in /etc/piler/piler-nginx.conf according to /etc/piler/piler-nginx.conf.dist

* If using the CUSTOM_CSS variable, then be sure to fix it like

  $config['CUSTOM_CSS'] = '<link rel="stylesheet" href="/path/to/my.css">';

* Added i18n tool, see /usr/libexec/piler/translate.php

* Database schema change. Be sure to run the upgrade sql script on
  all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
    mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
  done

* Added support for both session expiry and session inactivity, see
  SESSION_INACTIVITY_TIMEOUT and SESSION_EXPIRY_TIMEOUT in config.php

* Added support for Cloudflare Turnstile, a smart captcha alternative,
  see https://www.cloudflare.com/products/turnstile/ for more.

* Added OKTA oidc authentication support

* Added linkedin authentication support



1.7.3:

* Added imap connector utility

* Removed domain_user table and its references

* Improved Content-Security-Policy settings to disable 'unsafe-inline' javascript and CSS

* Rewrite uri changes

* Added support for hcaptcha, see https://www.hcaptcha.com/ for more.
  Removed deprecated secureimage project

* Be sure to update the rewrite rules and Content-Security-Policy settings
  in /etc/piler/piler-nginx.conf according to /etc/piler/piler-nginx.conf.dist


1.7.2:

* Rewrite uri changes. Be sure to update the rewrite rules in
  /etc/piler/piler-nginx.conf according to /etc/piler/piler-nginx.conf.dist

* Removed Auth0 support

* Fixed a parser issue causing random segfaults

* Installer scripts writes variables for goss. See
  https://github.com/jsuto/piler-examples/tree/master/examples/goss for more



1.7.1:

* imapfetch.py can call pilerimport to process a downloaded batch of emails
  with the -b option, eg. imapfetch.py -W piler -s ... -u ... -p ... -b 200

* pilerexport can organize exported files within the zip file to directories
  <zip batch size> number of files go to a single directory.

  Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
    mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
  done

* smtp message restore feature supports specifying username and password,
  as well as encrypted communication with the remote smtp host.

  Set the following in /etc/piler/config-site.php to use authenticated smtp restore:

  $config['SMARTHOST_USER'] = 'someuser';
  $config['SMARTHOST_PASSWORD'] = 'somepassword';

* Removed obsoleted variable 'tcp_wrappers_enable' from piler.conf



1.7.0:

* "Archiving rules" renamed to "Exclusion rules", because the
  purpose of those rules is to discard or ignore the macthing
  emails and prevent them getting archived.

  Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
    mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
    mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) <<< "rename table archiving_rule to exclusion_rule"
  done

* Fixed message highlighting bug for text mime parts

* Manticore search is the preferred search engine, however,
  sphinx search is still supported.

* If using sphinxsearch, then
    a) add the following to /etc/piler/config-site.php:

    $config['DIR_SPHINX'] = '/var/piler/sphinx/';

    b) be sure to have the following in /etc/piler/piler.conf:

    sphinxdir=/var/piler/sphinx

    c) ln -sf /etc/piler/sphinx.conf /etc/piler/manticore.conf

* Added real time (rt) index support to manticore


1.6.3:

* Replaced legacy each() function calls with foreach()

* Introduced the CUSTOM_POST_MESSAGE_READ_FUNCTION hook

* Introduced the CUSTOM_PRE_MESSAGE_DOWNLOAD_FUNCTION hook

* Fixed reindex utility

* Fixed handling long email addresses

* Fixed the sign.php utility

* Updated s3 python utilities to support minio==7.1.5
  Be sure to upgrade the minio python client: pip3 install minio==7.1.5

* Added Keycloak authentication support

* Added support for storing the archived files in subdirs inside the given bucket.
  To enable this feature, set s3_use_subdirs=1 in /etc/piler/piler.conf,
  and restart the piler-s3 systemd service (AKA the s3up.py script).

  This feature creates 256 dirs in the bucket.

  The default behaviour is to keep all archived files in ther root of the bucket.

  Add s3_use_subdirs=0 to /etc/piler/piler.conf if you already use S3 backend
  to keep compatibility.

  Note that you should not use both s3_spread_in_buckets=1 and s3_use_subdirs=1.



1.6.2:

* timestamp signing sorts by 'id' column

* Various minor fixes and improvements

  Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
     mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
  done

* Added support for spreading the archived files in several S3 buckets.
  To enable this feature, set s3_spread_in_buckets=1 in /etc/piler/piler.conf,
  and restart the piler-s3 systemd service (AKA the s3up.py script).

  This feature creates 256 buckets per organization / customer.

  The default behaviour is to keep all archived files in a single bucket for the
  given organization / customer.

  Add s3_spread_in_buckets=0 to /etc/piler/piler.conf if you already use S3 backend
  to keep compatibility.


1.6.1:

* Added add_header X-Frame-Options SAMEORIGIN always; to nginx config

* Introduced sender table

  Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
     mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
  done

* imapfetch.py uses ssl/tls by default. Use the --no-ssl option to revert to cleartext

  It also uses imapclient python module, so be sure to install it:

  pip install imapclient


1.6.0:

* REST API endpoint is available via /api/v1/
  Eg. https://archive.yourdomain.com/api/v1/get_domains

* Rewrite uri changes. Be sure to update the rewrite rules in
  /etc/piler/piler-nginx.conf according to /etc/piler/piler-nginx.conf.dist

* Increased max_packet_size to 32M in /etc/piler/sphinx.conf

* Introduced the bucket feature

  Be sure to run /usr/share/piler/db-upgrade.sql on all piler related databases, eg.

  for i in /var/piler/sphinx/piler $(find /var/piler/sphinx/* -type d); do
     mysql --defaults-file=/etc/piler/.my.cnf $(basename $i) < /usr/share/piler/db-upgrade.sql
  done

  See the video tutorial on the bucket feature at https://youtu.be/9U8C8u3r_Bw

* Added beacon to indicate which main index is to be used
  Run the following command to create the default value:

  while read -r d; do echo main1 > "${d}/MAIN"; done < <(find /var/piler/sphinx/* -type d)

* Removed the ldap_port column from ldap table
  The LDAP port should be part of the ldap hostname URI, eg. ldaps://ldap.host:1234

* If generating audit records is enabled, you may syslog the audit records to the mail log
  additionally by setting the following variable in config-site.php:

  $config['SYSLOG_AUDIT'] = 1;

  The idea is to allow you move audit records to ELK / EFK stack, Graylog or similar central
  logging system and make it searchable. Be sure to fix your syslog config to forward the
  mail log entries to the central logging system.

* Introduced new piler.conf variable: tls_min_version

  It sets the minimum TLS protocol version the piler-smtp daemon supports.

  Possible values:
  - TLSv1   (not recommended)
  - TLSv1.1 (not recommended)
  - TLSv1.2 (default)
  - TLSv1.3

* Added native support to authenticate against Azure AD
  See https://mailpiler.com/using-azure-ad-single-sign-on-sso/ for more.

* Added native support to authenticate against AWS Cognito
  See https://mailpiler.com/using-aws-cognito-single-sign-on-sso/ for more.

* Reindex utility should cleanup the temp files

* Improved the pilerexport zip writing performance

* The syslog entry is more concise for the piler components, eg.
  piler/smtp, piler/store, piler/webui, etc.

* If the Date: field contains some garbage return the current timestamp

* Introduced the max release feature. It's relevant to no-tenant
  installations only.

* SMTP protocol fix

* Parser fixes

* Emails having a NUL byte are redirected to the error directory



1.5.0:

* Install the python3-mysqldb package (Ubuntu)

  On centos7 and 8 run the following if you don't have this pip installed:
  pip3 install https://download.mailpiler.com/generic-local/mysqlclient-1.4.6-cp36-cp36m-linux_x86_64.whl

* Add the following entries to the cronjob of piler user on the worker nodes:

  */10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors
  2,22,42 * * * * /usr/libexec/piler/import.sh

* Add the following column to the customer_settings table, if it's not already there:

  alter table customer_settings add column if not exists `admin_password` varchar(128) default null;

* Introduced smtp acl list, and obsoleted tcp_wrappers

* Parse Sender: header and give it preference over From: header

* pilerimport puts legacy emails to a top level dir based on their Date: header

* pilerexport may digitally sign the exported zip file

  openssl genpkey -out /etc/piler/signkey.priv -algorithm rsa -pkeyopt rsa_keygen_bits:4096
  openssl rsa -in /etc/piler/signkey.priv -pubout -out /etc/piler/signkey.pub
  chown piler:piler /etc/piler/signkey.*

  sign: pilerexport -W customer1 -f some@user -z emails.zip -p /etc/piler/signkey.priv
  verify: openssl dgst -sha256 -verify /etc/piler/signkey.pub -signature emails.zip.sig emails.zip

  See the following for more:
    https://www.zimuel.it/blog/sign-and-verify-a-file-using-openssl
    https://opensource.com/article/19/6/cryptography-basics-openssl-part-2

* pilerexport may export EML files to a zip file. Use -z <zipfile>, eg.

  pilerexport -W customer1 -f some@user -z first100.zip

  Note that piler user must be able to create the zip file at the
  given path.

* Added RFC3161 timestamp support

  Run the following sql query on all piler related databases:

  alter table customer_settings add column `timestamp` varchar(6) default null;

* Preview body of email by hovering mouse over subject lines of the search results

* Added support for custom date format and timezone for users

  Run the following sql query on all piler related databases:

  alter table user_settings add column date_format char(12) default 'Y.m.d';
  alter table user_settings add column timezone varchar(32) default null;

* Last search is saved in a cookie, and restored when the user logs in again.

* Exclude any email address longer than 41 characters.
  SphinxQL breaks the query in case of any email 42+ characters long.

* Added support for zipped store directories.

  You may consolidate the zillions of .m files in /var/piler/store/<customerid>/
  to a series of zip files, eg.

  /var/piler/store/somecustomer/00/5f3/00 -> /var/piler/store/somecustomer/00/5f3/5f3_00.zip
  /var/piler/store/somecustomer/00/5f3/01 -> /var/piler/store/somecustomer/00/5f3/5f3_01.zip
  ...

  To enable the feature set consolidated_store=1 in /etc/piler/piler.conf
  Then run /usr/libexec/piler/compact-store-dir.sh 5f3 "somecustomer"

  Note that you may compact any top level dir except the last one which
  is currently used.

* Improved the encryption resilience by prepending the data to be encrypted with a block of garbage

* Switched from Blowfish encryption to AES-256. It's backward compatible.

  Fix the attachment table in all piler related databases:

  alter table attachment add column `e` char(1) default 'b';

* IMAP import over the gui

  To enable the feature for admin users set the following in /etc/piler/config-site.php:

  $config['ENABLE_IMPORT'] = 1;

  Also fix the import table in all piler related databases:

  alter table import add column `customer` varchar(255) default null;

  If you have a multinodes layout, then run the following command on the
  worker node which you want to run import jobs on (otherwise it won't
  process the import jobs):

  touch /etc/piler/IMPORT_HOST

* Discard short header only emails not having a Message-ID: line to prevent them
  filling up the error directory.


1.4.9:

* New customer form supports specifying the admin@local password
  for the customer at the 'Admin user password' input field.

* Support for sphinx-3.3.1

  If you upgrade piler, then be sure to set the following in /etc/piler/config-site.php
  to keep using the legacy full text search columns in sphinx:

  $config['SPHINX_STRICT_SCHEMA'] = 0;

  Note that enabling SPHINX_STRICT_SCHEMA (by setting it to 1) breaks the sphinx
  index data, and they must be recreated with reindex which takes some time.

* Added support for multiple master nodes

  Add the following line to /etc/piler/config-site.php on the current master/gui node
  if you have a multinode layout:

  $config['PRIMARY_MASTER'] = 1;


  To setup an additional, secondary master node set the following to config-site.php:

  $config['NODE_TYPE'] = MASTER;
  $config['PRIMARY_MASTER'] = 0;
  $config['DB_HOSTNAME'] = '<primary master>';
  $config['MASTER_NODE'] = '<primary master>';
  $memcached_server = ['<primary master>', 11211];

  In the above example the secondary master will use the mysql and memcached servers
  running on the primary master node.

* Fixed space projection calculation

* Added zipkin compatible tracing support for the GUI

* Don't create a default auditor@local account when creating a new tenant/customer

* Fixed unique constraint on accounting table:

-  UNIQUE (`date`, `email`)
+  UNIQUE (`date`, `email`, `server_id`)

* Fixed open_database() call to add mysqlhost parameter

* Fixed a typo in r.php

* Online users are written to and read from memcached exclusively

* Disable prometheus metrics by default. To enable set the following
  in /etc/piler/config-site.php:

  $config['ENABLE_PROMETHEUS_METRICS'] = 1;

  Also it's recommended to set some sort of access control to nginx config, eg.

        location = /metrics.php {
            # Add the IP-address of your Prometheus server
            allow 10.1.1.2;
            deny all;

            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_path_info;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            if (!-f $document_root$fastcgi_script_name) {
               return 404;
            }

            fastcgi_pass unix:PHP_FPM_SOCKET;
            fastcgi_index index.php;
            include fastcgi_params;
        }


1.4.8:

* Fixed a bug causing bogus customer info the smtp session

* Added piler-s3 systemctl service

* Updated cdn links in layout templates, and the matching nginx config

  Be sure to update the Content-Security-Policy definition:

  add_header Content-Security-Policy "default-src 'self' 'unsafe-inline' ; object-src 'none'; base-uri 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' code.jquery.com stackpath.bootstrapcdn.com; style-src 'self' 'unsafe-inline' stackpath.bootstrapcdn.com; font-src 'self' stackpath.bootstrapcdn.com; ";

* Introduced an improved mobile search template
  Set $config['ENABLE_MOBILE_PREVIEW'] = 1; in
  config-site.php to enable it.

* Fixed a bug in the health-check.sh script in a multinodes layout.
  It mistakenly reported a worker -> master probe result

* Introduced the Data Officer role

  You need to create the following table in all piler related databases:

  create table if not exists `deleted` (
    `id` bigint unsigned not null unique,
    `requestor` varchar(128) not null,
    `reason1` varchar(255) not null,
    `date1` int unsigned default 0,
    `approver` varchar(255) default null,
    `reason2` varchar(255) default null,
    `date2` int unsigned default 0,
    `deleted` tinyint(1) default -1,
    key (`id`),
    key (deleted)
  ) Engine=InnoDB;

  Add the following rewrite to the webserver config /etc/piler/piler-nginx.conf,
  then reload the webserver:

  rewrite /rejectremove.php /index.php?route=message/rejectremove;

* Using bootstrap 3.4.1 minified js from CDN

1.4.7:

* Fixed customer id check

* Fixed creating local users in multitenancy mode

* Added backup codes for Google Authenticator

  You need to create the following table in all piler related databases:

  create table if not exists `ga_backup_code` (
    `username` varchar(64) not null,
    `code` char(6) not null,
    `used` tinyint default 0
  ) Engine=InnoDB;

  create index ga_backup_code_idx1 on ga_backup_code(username);

* Added systemd service files to /usr/libexec/piler

* S3 stuff uses /var/piler/s3 directory

  mkdir /var/piler/s3
  chown piler:piler /var/piler/s3
  chmod 700 /var/piler/s3

* S3 upload script (s3up.py) upgraded to use python3.

  You need the minio python module to use it, eg. pip3 install minio

* imapfetch.py and pilerpurge.py uses /usr/bin/python3 as well

* Introduced the search history feature

  Set ENABLE_MEMCACHED in either /etc/piler/config-site.php or in the per customer settings, and be sure to install
  the php-memcached package. It puts the search terms to memcached to offers them in reverse order, latest on top.

* Fixed Google authenticator for ldap logins

* Introduced Auth0 support for Azure
  Set ENABLE_AUTH0 and the appropriate AUTH0 variables in either /etc/piler/config-site.php or in the per customer settings

* Default Sphinx version is set to 311 in /var/piler/www/config.php
  If you have 2.2.x version of sphinx, be sure to add the following line to /etc/piler/config-site.php:

  $config['SPHINX_VERSION'] = 220;

* Added tika and disk info to node health status output

* Introduced the default SPHINX_WORKER_LISTEN_ADDRESS variable to set the sphinx listen address on the worker nodes

  $config['SPHINX_WORKER_LISTEN_ADDRESS'] = '0.0.0.0';

1.4.6:

* The gui provides output for prometheus at /metrics

  Upgrade:
  - Add the following to piler's crontab on the worker nodes:
    10 * * * * /usr/bin/find /var/piler/error -type f | wc -l > /var/piler/stat/errors


1.4.5:

* Introduced new config variable USE_SMTP_GATEWAY


1.4.4:

* Fixed ldap authentication bug (1f828af)

* ldap table has new field (ldap_port) defaults to 389 (b094020)

  Run the following sql statement on all piler related databases:
     alter table ldap add column `ldap_port` int default 389;